Oct 30 18:32:17.394559 ip-10-0-131-155 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:32:17.760421 ip-10-0-131-155 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:17.760421 ip-10-0-131-155 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:32:17.760421 ip-10-0-131-155 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:17.760954 ip-10-0-131-155 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:17.763968 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763843    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.763968 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763969    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763972    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763976    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763979    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763981    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763985    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763987    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763990    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763993    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763995    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.763998    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764000    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764003    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764006    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764008    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764010    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764013    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764016    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764018    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764021    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.764030 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764023    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764026    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764028    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764031    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764033    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764036    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764042    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764046    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764048    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764051    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764054    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764063    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764066    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764069    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764072    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764074    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764077    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764079    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.764495 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764083    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764087    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764090    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764093    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764096    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764098    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764101    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764104    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764106    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764109    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764111    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764113    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764116    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764118    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764121    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764124    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764127    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764130    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764133    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764135    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.764910 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764138    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764140    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764142    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764145    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764147    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764150    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764152    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764155    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764157    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764160    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764165    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764168    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764170    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764173    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764175    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764178    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764180    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764183    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764185    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.765383 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764188    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764190    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764193    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764195    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764198    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764200    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764203    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764206    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764209    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764212    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764215    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764218    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764220    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764223    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764225    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764228    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764230    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764232    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764235    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764237    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.765847 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764240    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764242    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764245    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764247    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764249    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764253    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.764255    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765018    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765025    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765030    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765033    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765036    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765039    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765041    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765044    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765047    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765049    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765052    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765055    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.766320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765058    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765061    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765064    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765067    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765070    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765072    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765075    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765077    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765080    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765082    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765085    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765087    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765090    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765092    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765095    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765097    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765101    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765106    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765109    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.766761 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765112    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765115    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765117    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765120    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765122    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765125    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765127    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765130    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765132    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765135    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765137    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765140    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765143    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765146    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765149    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765151    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765154    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765156    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765160    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765162    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.767211 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765165    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765167    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765170    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765172    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765175    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765177    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765180    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765182    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765185    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765187    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765190    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765192    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765195    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765197    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765200    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765202    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765204    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765207    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765209    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765212    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.767693 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765215    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765217    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765220    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765222    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765225    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765228    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765230    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765233    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765236    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765238    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765241    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765244    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765246    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765248    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765251    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765253    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765256    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765258    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765260    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.768161 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765263    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765265    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765268    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765270    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765273    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765275    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765277    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765280    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765282    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765300    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765304    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765307    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765310    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765312    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765315    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765317    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765796    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765802    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765805    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765808    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.768636 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765810    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765813    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765816    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765818    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765821    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765824    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765827    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765829    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765832    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765834    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765837    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765839    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765842    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765844    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765847    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765849    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765852    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765855    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765858    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.769105 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765860    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765863    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765866    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765868    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765871    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765873    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765875    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765878    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765881    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765883    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765886    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765888    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765891    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765893    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765896    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765898    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765903    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765906    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765909    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765911    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.769548 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765914    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765916    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765919    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765921    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765924    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765926    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765928    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765931    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765933    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765936    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765938    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765941    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765943    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765946    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765948    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765951    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765953    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765955    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765958    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765960    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.770044 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765963    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765966    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765968    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765971    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765973    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765976    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765978    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765981    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765985    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765987    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765990    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765993    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765995    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.765997    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766000    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766003    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766005    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766008    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766010    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766013    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.770519 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766015    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766018    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766020    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766022    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766025    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766027    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766030    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766033    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766037    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766040    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766043    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766046    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766049    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766052    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766054    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766057    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766060    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766063    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766066    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.770982 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766070    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766075    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766078    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.766081    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.767978    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.767989    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.767995    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768000    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768005    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768008    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768012    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768016    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768019    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768022    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768025    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768028    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768031    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768034    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768037    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768040    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768043    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768046    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768050    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768053    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:32:17.771475 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768056    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768059    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768062    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768066    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768069    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768072    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768075    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768078    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768081    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768085    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768088    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768090    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768094    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768097    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768100    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768103    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768106    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768109    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768113    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768116    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768119    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768122    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768124    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768128    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:32:17.772026 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768131    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768134    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768137    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768140    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768142    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768145    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768148    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768150    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768155    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768157    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768161    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768164    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768168    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768171    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768173    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768176    2576 flags.go:64] FLAG: --help="false"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768179    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768182    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768185    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768188    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768191    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768194    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768197    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:32:17.772607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768200    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768203    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768206    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768208    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768211    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768214    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768216    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768219    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768222    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768225    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768227    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768230    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768233    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768235    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768240    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768243    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768246    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768250    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768253    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768256    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768259    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768262    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768266    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768269    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768272    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:32:17.773124 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768275    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768278    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768281    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768295    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768298    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768301    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768304    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768314    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768317    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768320    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768323    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768326    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768328    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768331    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768334    2576 flags.go:64] FLAG: --port="10250"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768336    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768339    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-064129740e07dcfea"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768342    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768345    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768348    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768351    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768354    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768357    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768360    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768365    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768369    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:32:17.773820 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768372    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768375    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768378    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768381    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768383    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768386    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768389    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768392    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768395    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768398    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768401    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768404    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768406    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768409    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768412    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768415    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768418    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768421    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768423    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768428    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768431    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768434    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768438    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768441    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768443    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:32:17.774468 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768446    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768449    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768451    2576 flags.go:64] FLAG: --v="2"
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768455    2576 flags.go:64] FLAG: --version="false"
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768459    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768464    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768468    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768594    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768598    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768601    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768603    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768606    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768609    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768611    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768615    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768618    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768620    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768623    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768625    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768628    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768630    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.775067 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768633    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768635    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768638    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768641    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768644    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768646    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768649    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768652    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768654    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768657    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768659    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768662    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768664    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768667    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768669    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768672    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768675    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768677    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768679    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768683    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.775575 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768685    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768688    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768690    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768693    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768695    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768699    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768701    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768703    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768706    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768708    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768710    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768713    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768715    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768718    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768720    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768723    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768725    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768728    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768731    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768733    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.776038 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768735    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768738    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768740    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768742    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768745    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768747    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768749    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768752    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768754    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768757    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768759    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768761    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768764    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768769    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768773    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768776    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768780    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768783    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.776533 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768785    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768788    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768791    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768793    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768796    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768799    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768803    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768806    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768808    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768811    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768813    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768816    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768818    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768821    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768823    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768826    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768828    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768831    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768833    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768836    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.776974 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768838    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768841    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768843    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768845    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768848    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768850    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768853    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768855    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768858    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768860    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768864    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768866    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768869    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.768894    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768929    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768934    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.777447 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768937    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768940    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768943    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768945    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768948    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768950    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768953    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768956    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768958    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768961    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768963    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768965    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768968    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768970    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768973    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768976    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768978    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768980    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768983    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.777812 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768985    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768988    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768990    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768992    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768995    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.768997    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769000    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769004    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769006    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769009    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769011    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769014    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769016    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769020    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769024    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769027    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769029    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769032    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769034    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.778245 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769037    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769040    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769042    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769046    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769049    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769052    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769054    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769057    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769059    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769062    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769065    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769068    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769070    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769073    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769076    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769078    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769081    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769083    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769086    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.778806 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769088    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769092    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769095    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769097    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769100    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769102    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769105    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769107    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769110    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769113    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769115    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769118    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769120    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769123    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769125    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769128    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769131    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769133    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769135    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769138    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.779240 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769141    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769144    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769146    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769150    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769152    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769155    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769157    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769159    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769162    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769165    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769167    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769170    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769172    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769176    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769178    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769181    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769184    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769186    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769189    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769191    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.779708 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769193    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769196    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769198    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769201    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769203    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769206    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.769209    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.776727    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.776742    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.776755    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.776759    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776813    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776818    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776822    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776825    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776828    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776830    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776833    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776836    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776838    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.780199 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776841    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776843    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776846    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776848    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776851    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776854    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776856    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776858    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776861    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776863    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776866    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776868    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776871    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776874    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776877    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776879    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776881    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776884    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776887    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.780668 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776891    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776895    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776898    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776903    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776907    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776911    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776913    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776916    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776919    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776922    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776927    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776930    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776932    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776935    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776937    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776940    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776942    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776944    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.781091 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776947    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776949    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776952    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776954    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776957    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776959    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776962    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776964    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776966    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776969    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776971    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776974    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776977    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776979    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776982    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776985    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776988    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776990    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776993    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776995    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.781569 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.776998    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777000    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777003    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777005    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777008    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777010    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777012    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777015    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777017    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777019    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777022    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777024    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777026    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777029    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777031    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777034    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777036    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777039    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777042    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777044    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.782360 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777046    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777049    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777051    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777054    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777056    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777059    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777061    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777064    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777066    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777069    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777071    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777074    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777076    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777078    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777081    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777083    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777086    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777088    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.783075 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777090    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.777116    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777249    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777254    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777257    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777260    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777263    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777265    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777268    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777270    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777272    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777275    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777277    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777280    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777282    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777302    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.783507 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777306    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777310    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777312    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777315    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777317    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777320    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777323    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777326    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777329    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777331    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777334    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777337    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777339    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777342    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777344    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777346    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777349    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777351    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777354    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777356    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.783936 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777359    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777361    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777365    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777368    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777371    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777377    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777379    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777381    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777384    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777386    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777389    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777391    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777394    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777396    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777399    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777401    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777404    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777406    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.784423 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777409    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777414    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777416    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777418    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777421    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777423    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777426    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777428    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777432    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777435    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777438    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777440    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777443    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777445    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777448    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777450    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777453    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777456    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777459    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777461    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.784900 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777464    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777466    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777468    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777471    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777473    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777476    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777478    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777481    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777483    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777485    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777488    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777490    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777492    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777496    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777499    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777501    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777503    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777506    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777508    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.785388 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777511    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777513    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777516    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777518    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777520    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777522    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777525    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777527    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777530    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777532    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777534    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777537    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777539    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777542    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.777565    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777595    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:17.785916 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777599    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777602    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777604    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777609    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777612    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777614    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777617    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777619    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777622    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777624    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777628    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777631    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777634    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777636    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777639    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777641    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777643    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777646    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777648    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:17.786310 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777651    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777654    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777658    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777661    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777664    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777667    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777670    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777673    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777676    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777678    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777681    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777683    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777686    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777688    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777691    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777693    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777697    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777700    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777702    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777705    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:17.786740 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777707    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777710    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777712    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777716    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777718    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777720    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777723    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777726    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777728    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777731    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777733    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777735    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777738    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777741    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777743    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777746    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777748    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777750    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777753    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:17.787232 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777755    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777758    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777760    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777762    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777765    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777770    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777772    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777774    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777778    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777781    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777783    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777786    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777788    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777791    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777793    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777798    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777800    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777803    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:17.787729 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777805    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777808    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777810    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777813    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777816    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777818    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777821    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777824    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777826    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777829    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777831    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777834    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777836    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777839    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777841    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777844    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777846    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777848    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777851    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777853    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:17.788167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777856    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777858    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777861    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777864    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777868    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777871    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777874    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:17.777876    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.777924    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.780264    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.781321    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.781411    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.781718    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.787412    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:17.788689 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.788522    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:17.798918 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.798904    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:32:17.805282 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.805266    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 18:32:17.806179 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.806152    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:32:17.809743 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.809722    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7d3c7a3-4135-4ad8-8d72-d2a259ab25b9:/dev/nvme0n1p3 f6e0f408-4753-4d4f-8d02-45ad59897843:/dev/nvme0n1p4]
Oct 30 18:32:17.809815 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.809741    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:32:17.814787 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.814665    2576 manager.go:217] Machine: {Timestamp:2026-10-30 18:32:17.813158911 +0000 UTC m=+0.321309680 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119408 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b47f58ab6e323e34eae82abc2796 SystemUUID:ec24b47f-58ab-6e32-3e34-eae82abc2796 BootID:86cbab31-211f-4f83-8489-c00481e33aa3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:c5:41:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:c5:41:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:f0:c6:c7:4c:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:32:17.814787 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.814779    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:32:17.814936 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.814870    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:32:17.817766 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817741    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:32:17.817889 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817767    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-155.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:32:17.817937 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817891    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:32:17.817937 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817899    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:32:17.817988 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817941    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:17.817988 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.817958    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:17.819170 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.819159    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:32:17.819987 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.819970    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:32:17.823877 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.823865    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:32:17.823996 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.823986    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:32:17.824031 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824014    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:32:17.824031 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824028    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:32:17.824099 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824040    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:32:17.824099 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824048    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:32:17.824099 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824056    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:32:17.824449 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.824432    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:17.828149 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.828065    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:32:17.834140 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.834124    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:17.835088 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835071    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:32:17.835205 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835103    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:32:17.835255 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835245    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:32:17.835302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835260    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:32:17.835302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835267    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:32:17.835302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835272    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:32:17.835302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835277    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:32:17.835302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835283    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:32:17.836006 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.835997    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:32:17.836035 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.836007    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:32:17.836035 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.836016    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:32:17.836035 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.836021    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:32:17.836035 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.836029    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:32:17.836147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.836038    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:32:17.837665 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.837656    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:32:17.837700 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.837666    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:32:17.841543 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.841528    2576 server.go:1263] "Started kubelet"
Oct 30 18:32:17.841623 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.841583    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:32:17.841679 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.841611    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:32:17.841737 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.841671    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:32:17.842342 ip-10-0-131-155 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:32:17.842800 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.842768    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:32:17.845751 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.845733    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:32:17.845861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.845824    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-155.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:32:17.845938 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.845901    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-155.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:32:17.845988 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.845932    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:32:17.848556 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.848538    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:32:17.852454 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.852440    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:32:17.853043 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853028    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:32:17.853162 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.853140    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:17.853814 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853793    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:32:17.853913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853842    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:32:17.853913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853789    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:32:17.853913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853863    2576 factory.go:55] Registering systemd factory
Oct 30 18:32:17.854062 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.853945    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:32:17.854101 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854069    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:32:17.854101 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854077    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:32:17.854161 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854142    2576 factory.go:153] Registering CRI-O factory
Oct 30 18:32:17.854161 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854154    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:32:17.854226 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854216    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:32:17.854262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854237    2576 factory.go:103] Registering Raw factory
Oct 30 18:32:17.854262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.854251    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:32:17.855424 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.855405    2576 manager.go:319] Starting recovery of all containers
Oct 30 18:32:17.857335 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.857211    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-155.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:32:17.858811 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.857543    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f681eba0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.84149904 +0000 UTC m=+0.349649807,LastTimestamp:2026-10-30 18:32:17.84149904 +0000 UTC m=+0.349649807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.863503 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.863474    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:32:17.868643 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.868618    2576 manager.go:324] Recovery completed
Oct 30 18:32:17.872239 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.872226    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:17.873906 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.873890    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:17.873963 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.873918    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:17.873963 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.873927    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:17.874449 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.874436    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:32:17.874449 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.874447    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:32:17.874539 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.874466    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:32:17.874539 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.874436    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:17.875144 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.875129    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:17.875214 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.875151    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:17.875214 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.875161    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:17.876632 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.876617    2576 policy_none.go:50] "Start"
Oct 30 18:32:17.876632 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.876632    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:32:17.876730 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.876642    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:32:17.877423 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.877347    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f87063bd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873904573 +0000 UTC m=+0.382055340,LastTimestamp:2026-10-30 18:32:17.873904573 +0000 UTC m=+0.382055340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.878658 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.878645    2576 policy_none.go:44] "Start"
Oct 30 18:32:17.889635 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.889432    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f870a9d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873922515 +0000 UTC m=+0.382073284,LastTimestamp:2026-10-30 18:32:17.873922515 +0000 UTC m=+0.382073284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.899946 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.899888    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f870cc81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873931393 +0000 UTC m=+0.382082164,LastTimestamp:2026-10-30 18:32:17.873931393 +0000 UTC m=+0.382082164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.906152 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.906088    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-155.us-east-2.compute.internal.18e36247f87063bd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f87063bd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873904573 +0000 UTC m=+0.382055340,LastTimestamp:2026-10-30 18:32:17.87514029 +0000 UTC m=+0.383291057,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.908065 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.908045    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7fgs"
Oct 30 18:32:17.911618 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.911550    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-155.us-east-2.compute.internal.18e36247f870a9d3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f870a9d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873922515 +0000 UTC m=+0.382073284,LastTimestamp:2026-10-30 18:32:17.875155245 +0000 UTC m=+0.383306013,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.911808 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.911779    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916555    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.916578    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916586    2576 server.go:87] "Starting device plugin registration server"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916795    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916806    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916923    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916989    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.916997    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.918327    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.918367    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.921874    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7fgs"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.926152    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-155.us-east-2.compute.internal.18e36247f870cc81\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-155.us-east-2.compute.internal.18e36247f870cc81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-155.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-155.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-155.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:17.873931393 +0000 UTC m=+0.382082164,LastTimestamp:2026-10-30 18:32:17.87516622 +0000 UTC m=+0.383316989,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-155.us-east-2.compute.internal,}"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.930274    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.931544    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.931573    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.931597    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:32:17.934262 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:17.931638    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:32:17.940182 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:17.940168    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:18.017509 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.017461    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:18.018217 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.018202    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:18.018311 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.018234    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:18.018311 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.018245    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:18.018311 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.018275    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.028956 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.028940    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.029003 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.028959    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-155.us-east-2.compute.internal\": node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.032484 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.032466    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"]
Oct 30 18:32:18.032549 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.032526    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:18.033228 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.033213    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:18.033327 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.033245    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:18.033327 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.033261    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:18.035502 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.035487    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:18.035637 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.035623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.035714 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.035649    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:18.036206 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036190    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:18.036271 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036224    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:18.036271 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036240    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:18.036271 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036248    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:18.036271 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036267    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:18.036408 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.036277    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:18.038401 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.038383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.038476 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.038425    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:18.039126 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.039110    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:18.039171 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.039140    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:18.039171 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.039156    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:18.063584 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.063564    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.075907 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.075888    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.082298 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.082268    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.154599 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.154570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.154690 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.154621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.154690 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.154639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.164176 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.164155    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.254990 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.254967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.255084 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.255021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.255084 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.255070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.255153 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.255075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.255153 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.255136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.255212 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.255157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.264932 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.264912    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.365708 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.365657    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.379844 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.379824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.386645 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.386628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.466749 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.466721    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.567189 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.567160    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.667767 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.667715    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:18.684184 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.684160    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:18.754263 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.754237    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.768611 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.768590    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:32:18.770646 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.770629    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.780812 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.780796    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:32:18.780937 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.780918    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:32:18.780992 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.780937    2576 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a97b000890fe34dfd935cea991184764-22fd132c3e50da3b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.131.155:42984->77.113.125.184:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:18.825071 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.825045    2576 apiserver.go:52] "Watching apiserver"
Oct 30 18:32:18.835437 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.835406    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:32:18.836210 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.836179    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9lr9k","openshift-dns/node-resolver-prl4g","openshift-image-registry/node-ca-fqpv4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-lg2zm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9","openshift-multus/network-metrics-daemon-c5n4z","kube-system/konnectivity-agent-nhsc4","kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-q9pks","openshift-network-diagnostics/network-check-target-w7prc","openshift-network-operator/iptables-alerter-tnzkp","openshift-cluster-node-tuning-operator/tuned-wpghw","openshift-multus/multus-g2z97"]
Oct 30 18:32:18.839057 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.839032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.841128 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.841107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.843632 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.843613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.845713 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.845688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.845832 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.845770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:18.847918 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.847898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.850280 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.850250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.850396 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.850347    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:18.850396 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.850389    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:32:18.850526 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.850440    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\""
Oct 30 18:32:18.850526 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.850512    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\""
Oct 30 18:32:18.851628 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.851603    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.852168 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852150    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:32:18.852168 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852165    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\""
Oct 30 18:32:18.852276 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852170    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.852276 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852168    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.852276 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852153    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:32:18.852276 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.852607 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852591    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:32:18.852648 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.852621    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.853419 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.853401    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:32:18.855059 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.855040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.857984 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.857965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:18.858062 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.858029    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:18.858472 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.858512 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858512 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858601 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858601 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.858601 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.858747 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.858747 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.858747 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858747 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6pz\" (UniqueName: \"kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.858971 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.858996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t777l\" (UniqueName: \"kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncxb\" (UniqueName: \"kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828xl\" (UniqueName: \"kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rgf\" (UniqueName: \"kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.859695 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859712    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.860157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.859840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.860512 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.860251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:18.862547 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.862531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.865268 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.865250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.928557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.928502    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:18 +0000 UTC" deadline="2028-06-05 14:28:40.297226743 +0000 UTC"
Oct 30 18:32:18.928557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.928553    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14011h56m21.368675617s"
Oct 30 18:32:18.952645 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952626    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.952645 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952644    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.952806 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952671    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:32:18.952863 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952809    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.952863 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952822    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.952863 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952825    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.952992 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952884    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:32:18.952992 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.952893    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\""
Oct 30 18:32:18.953060 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.953038    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\""
Oct 30 18:32:18.954778 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954760    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.954895 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954821    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.954895 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954828    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:32:18.954895 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954838    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:32:18.954895 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954860    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954919    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954929    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954951    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954978    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955000    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955002    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.954921    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955041    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955043    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:32:18.955139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955049    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:32:18.955500 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.955397    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:32:18.959994 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.959972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960104 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960104 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960104 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960115    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960225 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncxb\" (UniqueName: \"kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.960440 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rgf\" (UniqueName: \"kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.960659 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr5m\" (UniqueName: \"kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960790    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.960843 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.960977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961022    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961147 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961196    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbvm\" (UniqueName: \"kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961515    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t777l\" (UniqueName: \"kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7hkd\" (UniqueName: \"kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.961676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961690    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828xl\" (UniqueName: \"kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961922    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.961992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.962032    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.962111    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:19.462079409 +0000 UTC m=+1.970230184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:18.962577 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.963400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962586    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.962684    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:18.962890    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:19.462866448 +0000 UTC m=+1.971017203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.962976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6pz\" (UniqueName: \"kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963003    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.964160 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963396    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963679    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.963798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:18.964934 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.964909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.965189 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.965106    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:18.974778 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.974758    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:32:18.975208 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.975186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncxb\" (UniqueName: \"kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:18.976279 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.976256    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t777l\" (UniqueName: \"kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:18.976408 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.976383    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828xl\" (UniqueName: \"kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:18.988383 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.988366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6pz\" (UniqueName: \"kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:18.989369 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.989348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:18.989624 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.989605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rgf\" (UniqueName: \"kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:18.998816 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:18.998799    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p52wg"
Oct 30 18:32:19.003887 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.003872    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:19.029424 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.029407    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p52wg"
Oct 30 18:32:19.031601 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.031586    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:19.042666 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.042640    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabecda5632cef959a155a21c4c95263.slice/crio-cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2 WatchSource:0}: Error finding container cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2: Status 404 returned error can't find the container with id cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2
Oct 30 18:32:19.049150 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.049134    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:32:19.053320 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.053265    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210591172398def826e79b175b0190f4.slice/crio-c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4 WatchSource:0}: Error finding container c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4: Status 404 returned error can't find the container with id c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4
Oct 30 18:32:19.064043 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064115 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.064115 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064181 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064181 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064181 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.064262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.064262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064262 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:19.064376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064518    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr5m\" (UniqueName: \"kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064697    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.064728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064733    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064847    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064904    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064972    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064982    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.064959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbvm\" (UniqueName: \"kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065080    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.065600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hkd\" (UniqueName: \"kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065335    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065430    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.066488 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.065687    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.067239 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.067220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.067551 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.067537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.077395 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.077380    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:19.077463 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.077398    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:19.077463 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.077407    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:19.077463 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.077448    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:19.577435847 +0000 UTC m=+2.085586635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:19.079342 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.079323    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr5m\" (UniqueName: \"kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.081032 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.081010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hkd\" (UniqueName: \"kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.081202 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.081187    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbvm\" (UniqueName: \"kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.198419 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.198366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:19.204384 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.204370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:32:19.208031 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.208009    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb49e957_8fde_4701_bbb0_b718a6acd947.slice/crio-854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421 WatchSource:0}: Error finding container 854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421: Status 404 returned error can't find the container with id 854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421
Oct 30 18:32:19.212379 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.212359    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc98e36_faf2_4b8a_b2b5_b1d295c01fc2.slice/crio-523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049 WatchSource:0}: Error finding container 523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049: Status 404 returned error can't find the container with id 523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049
Oct 30 18:32:19.222343 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.222326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:32:19.230646 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.230623    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa09131_1c47_4d63_82d2_c5174c60fd49.slice/crio-91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051 WatchSource:0}: Error finding container 91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051: Status 404 returned error can't find the container with id 91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051
Oct 30 18:32:19.243211 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.243190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:32:19.245347 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.245334    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:19.251327 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.251271    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd961e0b7_299c_413d_8206_d7e9398e76b2.slice/crio-b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24 WatchSource:0}: Error finding container b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24: Status 404 returned error can't find the container with id b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24
Oct 30 18:32:19.255390 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.255369    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb2a473_98fc_4918_a07e_fb7d4edfddb8.slice/crio-213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391 WatchSource:0}: Error finding container 213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391: Status 404 returned error can't find the container with id 213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391
Oct 30 18:32:19.262727 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.262711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:32:19.271397 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.271377    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a74cbcd_a5c8_474b_8206_6f4b7e17b416.slice/crio-dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b WatchSource:0}: Error finding container dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b: Status 404 returned error can't find the container with id dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b
Oct 30 18:32:19.274571 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.274558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:32:19.282810 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.282782    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd741136f_7a0a_479a_84ea_d74bdc485189.slice/crio-7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e WatchSource:0}: Error finding container 7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e: Status 404 returned error can't find the container with id 7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e
Oct 30 18:32:19.293613 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.293595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:32:19.302074 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.302056    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4411e7a_f2fb_4f3a_854c_f19b0e70b420.slice/crio-92bc42e76c87bfe827d017c9b683239a287fd9a9027d70530bad69abdee9d6f4 WatchSource:0}: Error finding container 92bc42e76c87bfe827d017c9b683239a287fd9a9027d70530bad69abdee9d6f4: Status 404 returned error can't find the container with id 92bc42e76c87bfe827d017c9b683239a287fd9a9027d70530bad69abdee9d6f4
Oct 30 18:32:19.306092 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.306078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2z97"
Oct 30 18:32:19.314275 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:19.314257    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3392f519_fd88_4b83_bda6_f7489c047423.slice/crio-519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e WatchSource:0}: Error finding container 519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e: Status 404 returned error can't find the container with id 519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e
Oct 30 18:32:19.469437 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.469394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:19.469437 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.469428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:19.469537 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.469526    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:19.469537 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.469528    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:19.469594 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.469567    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:20.469556251 +0000 UTC m=+2.977707018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:19.469594 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.469580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:20.46957459 +0000 UTC m=+2.977725344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:19.670817 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:19.670782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:19.671019 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.670954    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:19.671019 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.670974    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:19.671019 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.670986    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:19.671179 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:19.671043    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:20.67102575 +0000 UTC m=+3.179176517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:20.989190 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:20.989074    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2z97" event={"ID":"3392f519-fd88-4b83-bda6-f7489c047423","Type":"ContainerStarted","Data":"519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e"}
Oct 30 18:32:20.997572 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:20.997549    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:21.001257 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.001233    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpghw" event={"ID":"a4411e7a-f2fb-4f3a-854c-f19b0e70b420","Type":"ContainerStarted","Data":"92bc42e76c87bfe827d017c9b683239a287fd9a9027d70530bad69abdee9d6f4"}
Oct 30 18:32:21.014984 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.014912    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhsc4" event={"ID":"3fb2a473-98fc-4918-a07e-fb7d4edfddb8","Type":"ContainerStarted","Data":"213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391"}
Oct 30 18:32:21.023600 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.023513    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9" event={"ID":"d961e0b7-299c-413d-8206-d7e9398e76b2","Type":"ContainerStarted","Data":"b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24"}
Oct 30 18:32:21.028101 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.028059    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fqpv4" event={"ID":"2fa09131-1c47-4d63-82d2-c5174c60fd49","Type":"ContainerStarted","Data":"91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051"}
Oct 30 18:32:21.034471 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.034444    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-prl4g" event={"ID":"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2","Type":"ContainerStarted","Data":"523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049"}
Oct 30 18:32:21.048845 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.048820    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421"}
Oct 30 18:32:21.067029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.067003    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerStarted","Data":"c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4"}
Oct 30 18:32:21.069080 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.069057    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnzkp" event={"ID":"d741136f-7a0a-479a-84ea-d74bdc485189","Type":"ContainerStarted","Data":"7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e"}
Oct 30 18:32:21.071908 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.071884    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerStarted","Data":"dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b"}
Oct 30 18:32:21.074518 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.074460    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:20 +0000 UTC" deadline="2028-08-09 09:11:54.734606552 +0000 UTC"
Oct 30 18:32:21.074606 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.074519    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15566h39m33.660091328s"
Oct 30 18:32:21.077723 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.077703    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"dabecda5632cef959a155a21c4c95263","Type":"ContainerStarted","Data":"cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2"}
Oct 30 18:32:21.518730 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.518700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:21.518888 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.518750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:21.518888 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.518871    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:21.519004 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.518925    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:23.518909291 +0000 UTC m=+4.986262639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:21.519338 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.519318    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:21.519449 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.519373    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:23.519357178 +0000 UTC m=+4.986710510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:21.721476 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.721444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:21.721648 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.721616    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:21.721648 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.721638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:21.721748 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.721660    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:21.721748 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.721713    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:23.721695593 +0000 UTC m=+5.189048939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:21.973457 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.973336    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:21.973619 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.973529    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:21.974096 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.973986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:21.974199 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.974110    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:21.974259 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:21.974209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:21.974346 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:21.974297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:22.075637 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:22.075574    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:20 +0000 UTC" deadline="2028-07-14 23:47:27.93400466 +0000 UTC"
Oct 30 18:32:22.075637 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:22.075635    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14957h15m5.85837388s"
Oct 30 18:32:23.535326 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.535299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:23.535719 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.535342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:23.535719 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.535469    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:23.535719 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.535519    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:27.535501607 +0000 UTC m=+9.002854953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:23.535953 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.535932    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:23.536018 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.535994    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:27.535977435 +0000 UTC m=+9.003330768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:23.736594 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.736551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:23.736752 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.736705    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:23.736752 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.736722    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:23.736752 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.736736    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:23.736903 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.736798    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:27.736779183 +0000 UTC m=+9.204132522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:23.973476 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.973258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:23.973604 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.973584    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:23.973995 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.973954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:23.974108 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.974049    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:23.974169 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:23.974137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:23.974231 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:23.974219    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:25.095893 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.095813    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhsc4" event={"ID":"3fb2a473-98fc-4918-a07e-fb7d4edfddb8","Type":"ContainerStarted","Data":"583fc8d3dfa18c2ac34feb3b1374538c03f050bf2ed5cf0a40636fc0b58af00a"}
Oct 30 18:32:25.103052 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.103021    2576 generic.go:356] "Generic (PLEG): container finished" podID="210591172398def826e79b175b0190f4" containerID="ac284a4bb6fd3802542259a3ec8453625147486172246eb043cade75e6772593" exitCode=0
Oct 30 18:32:25.103200 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.103060    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerDied","Data":"ac284a4bb6fd3802542259a3ec8453625147486172246eb043cade75e6772593"}
Oct 30 18:32:25.110553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.110523    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="14f81edd5c5895968e62cd679a45c01ab2955d8016d900a8e58fbeeb820c5130" exitCode=0
Oct 30 18:32:25.110675 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.110597    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"14f81edd5c5895968e62cd679a45c01ab2955d8016d900a8e58fbeeb820c5130"}
Oct 30 18:32:25.114790 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.114065    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nhsc4" podStartSLOduration=1.616295982 podStartE2EDuration="6.11405137s" totalImagesPullingTime="4.497755388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:25.113341201 +0000 UTC m=+6.580694557" watchObservedRunningTime="2026-10-30 18:32:25.11405137 +0000 UTC m=+6.581404726"
Oct 30 18:32:25.115386 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.114982    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"dabecda5632cef959a155a21c4c95263","Type":"ContainerStarted","Data":"90a3f282cb587398a3f40f052eda6fb966888f7e822cb9973fa326d5281ce770"}
Oct 30 18:32:25.972904 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.972871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:25.973062 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:25.973007    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:25.973447 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.973423    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:25.973547 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:25.973524    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:25.973614 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:25.973599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:25.973702 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:25.973684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:26.131647 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:26.131610    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerStarted","Data":"92f6880441819cf1baddaa17e1726704e47d5b23e5899dc8bd4933b50eafd277"}
Oct 30 18:32:26.147922 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:26.147879    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" podStartSLOduration=7.147867034 podStartE2EDuration="7.147867034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:26.147792183 +0000 UTC m=+7.615145539" watchObservedRunningTime="2026-10-30 18:32:26.147867034 +0000 UTC m=+7.615220388"
Oct 30 18:32:26.148128 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:26.148067    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal" podStartSLOduration=7.148062744 podStartE2EDuration="7.148062744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:25.176801652 +0000 UTC m=+6.644155007" watchObservedRunningTime="2026-10-30 18:32:26.148062744 +0000 UTC m=+7.615416095"
Oct 30 18:32:26.186958 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:26.186934    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:26.187883 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:26.187859    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:27.136391 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.136355    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9" event={"ID":"d961e0b7-299c-413d-8206-d7e9398e76b2","Type":"ContainerStarted","Data":"ae1a4d1390f683a91ee7a243f0a526da7c5d459552e686bf8011ca63496329b8"}
Oct 30 18:32:27.139583 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.139497    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fqpv4" event={"ID":"2fa09131-1c47-4d63-82d2-c5174c60fd49","Type":"ContainerStarted","Data":"59c3496442f72130531e22aca01e777206abf6498281e32f9aacbe26bc583400"}
Oct 30 18:32:27.156050 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.155945    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fqpv4" podStartSLOduration=2.422511889 podStartE2EDuration="8.155928782s" totalImagesPullingTime="5.733416893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:27.15493733 +0000 UTC m=+8.622290688" watchObservedRunningTime="2026-10-30 18:32:27.155928782 +0000 UTC m=+8.623282137"
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.567067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.567148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.567304    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.567369    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.567352148 +0000 UTC m=+17.034705481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.567801    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:27.567903 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.567857    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.567841963 +0000 UTC m=+17.035195301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:27.769835 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.769150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:27.769835 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.769328    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:27.769835 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.769347    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:27.769835 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.769361    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:27.769835 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.769416    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.769399492 +0000 UTC m=+17.236752824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:27.972915 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.972856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:27.972915 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.972878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:27.973173 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.972978    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:27.973532 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.973357    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:27.973532 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:27.973409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:27.973532 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:27.973479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:28.141286 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:28.140293    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:32:28.316625 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:28.316594    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:32:28.967107 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:28.963478    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:32:28.316621161Z","UUID":"97ff1c74-a826-430c-aea8-fc08a56d5b68","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:32:28.967107 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:28.965355    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:32:28.967107 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:28.965378    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:32:29.143986 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.143946    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2z97" event={"ID":"3392f519-fd88-4b83-bda6-f7489c047423","Type":"ContainerStarted","Data":"86deb24dc8735d23ff139dc933eedda162b7db8a45c8d9f634a97aef6a0a3869"}
Oct 30 18:32:29.148405 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.148375    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9" event={"ID":"d961e0b7-299c-413d-8206-d7e9398e76b2","Type":"ContainerStarted","Data":"951da1043983dd7d62f961e7b37e78cc9623f88ada55258ce7e0ad952cd8df98"}
Oct 30 18:32:29.152349 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.151922    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-prl4g" event={"ID":"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2","Type":"ContainerStarted","Data":"4649a5dff40fe3f7edca49217c1bb59d5228fafa62605dcbabaf5fd58ef796e0"}
Oct 30 18:32:29.182380 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.182334    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g2z97" podStartSLOduration=2.157245661 podStartE2EDuration="10.182317546s" totalImagesPullingTime="8.025071885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:29.181820655 +0000 UTC m=+10.649174012" watchObservedRunningTime="2026-10-30 18:32:29.182317546 +0000 UTC m=+10.649670901"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.973317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:29.973446    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.973809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:29.973914    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:29.973974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:29.974134 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:29.974026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:30.157056 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:30.157021    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="00c863b0936cf8c4ee6881ab746fcab346e9f231bef57b548619afb704ee5386" exitCode=0
Oct 30 18:32:30.157622 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:30.157130    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"00c863b0936cf8c4ee6881ab746fcab346e9f231bef57b548619afb704ee5386"}
Oct 30 18:32:30.161446 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:30.161422    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9" event={"ID":"d961e0b7-299c-413d-8206-d7e9398e76b2","Type":"ContainerStarted","Data":"04462a33bf06c956d1b1ba0dfec9283e2625717f00ff2d4f9f7659f96b082e95"}
Oct 30 18:32:30.182193 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:30.182150    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-prl4g" podStartSLOduration=2.715441285 podStartE2EDuration="11.182135686s" totalImagesPullingTime="8.466694401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:29.203584248 +0000 UTC m=+10.670937604" watchObservedRunningTime="2026-10-30 18:32:30.182135686 +0000 UTC m=+11.649489041"
Oct 30 18:32:30.210025 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:30.209989    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9" podStartSLOduration=2.306888299 podStartE2EDuration="11.209977638s" totalImagesPullingTime="8.903089339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:30.20992315 +0000 UTC m=+11.677276501" watchObservedRunningTime="2026-10-30 18:32:30.209977638 +0000 UTC m=+11.677330991"
Oct 30 18:32:31.165273 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.165245    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpghw" event={"ID":"a4411e7a-f2fb-4f3a-854c-f19b0e70b420","Type":"ContainerStarted","Data":"177c2e4f1a40e51e0cf3688f795556896611bdbf101f7cfe8e6cb9fe4f29c604"}
Oct 30 18:32:31.167146 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.167125    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnzkp" event={"ID":"d741136f-7a0a-479a-84ea-d74bdc485189","Type":"ContainerStarted","Data":"25d6e5c6383ecd5455dad11aa1e6e931b5cdebe536957b81c56ba1fab221f233"}
Oct 30 18:32:31.200825 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.200775    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tnzkp" podStartSLOduration=3.591046938 podStartE2EDuration="12.200761474s" totalImagesPullingTime="8.609714536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:31.200226826 +0000 UTC m=+12.667580181" watchObservedRunningTime="2026-10-30 18:32:31.200761474 +0000 UTC m=+12.668114828"
Oct 30 18:32:31.200943 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.200868    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wpghw" podStartSLOduration=1.820707181 podStartE2EDuration="12.200862076s" totalImagesPullingTime="10.380154895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:31.18415875 +0000 UTC m=+12.651512105" watchObservedRunningTime="2026-10-30 18:32:31.200862076 +0000 UTC m=+12.668215429"
Oct 30 18:32:31.972931 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.972896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:31.972931 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.972931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:31.973189 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:31.972999    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:31.973189 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:31.973038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:31.973189 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:31.973159    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:31.973309 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:31.973238    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:32.171242 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:32.171204    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="d293e61dfb770ac07c3cacbff87235fbc4cb6b26cbd42b411a9f901f5b94d2c8" exitCode=0
Oct 30 18:32:32.171751 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:32.171292    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"d293e61dfb770ac07c3cacbff87235fbc4cb6b26cbd42b411a9f901f5b94d2c8"}
Oct 30 18:32:33.973099 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:33.972898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:33.973572 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:33.972988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:33.973572 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:33.973182    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:33.973572 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:33.973270    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:33.973572 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:33.972996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:33.973572 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:33.973368    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:34.178428 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:34.178380    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="a8eac129431479bcb1d722bd8194a8caccfdc42b1468ccb4ab419824ca9617d4" exitCode=0
Oct 30 18:32:34.178428 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:34.178425    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"a8eac129431479bcb1d722bd8194a8caccfdc42b1468ccb4ab419824ca9617d4"}
Oct 30 18:32:35.627685 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.627524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:35.627685 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.627677    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:35.628044 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.627738    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:51.627721724 +0000 UTC m=+33.095075057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:35.628044 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.627730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:35.628044 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.627824    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:35.628044 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.627880    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret podName:afe4c4f3-230a-4545-844b-fd72139f8e6d nodeName:}" failed. No retries permitted until 2026-10-30 18:32:51.627865301 +0000 UTC m=+33.095218634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret") pod "global-pull-secret-syncer-lg2zm" (UID: "afe4c4f3-230a-4545-844b-fd72139f8e6d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:35.828943 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.828880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:35.829157 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.828999    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:35.829157 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.829014    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:35.829157 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.829024    2576 projected.go:196] Error preparing data for projected volume kube-api-access-42v47 for pod openshift-network-diagnostics/network-check-target-w7prc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:35.829157 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.829071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47 podName:1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:51.829055358 +0000 UTC m=+33.296408708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-42v47" (UniqueName: "kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47") pod "network-check-target-w7prc" (UID: "1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:35.973226 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.973190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:35.973351 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.973190    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:35.973351 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:35.973312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:35.973351 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.973311    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:35.973488 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.973381    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:35.973529 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:35.973502    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:36.184980 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.184923    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:36.185322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185301    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb49e957-8fde-4701-bbb0-b718a6acd947" containerID="5b90f94b53a3c60b4a35d0b02adbd8310842582d2bcc75b9db4c1de0fc5571d8" exitCode=1
Oct 30 18:32:36.185402 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185373    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"a8e7c6af804ae4068a0875390a85085fbf02c38eeef2bdea7f612ea27f273743"}
Oct 30 18:32:36.185451 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185439    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"ed62d685e6d004836b7e9a10d461902c437d215834760c951f1ef310e55d5479"}
Oct 30 18:32:36.185503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185489    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"304da0b20784783aed78fa8403b060dde3db0bf067357950f86d8c01f29f6d54"}
Oct 30 18:32:36.185538 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185532    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"00785053404e55e56b4d4e1de5ad4a42bed6a7bc18b0d07cb918cd4647ffb0ac"}
Oct 30 18:32:36.185597 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185577    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerDied","Data":"5b90f94b53a3c60b4a35d0b02adbd8310842582d2bcc75b9db4c1de0fc5571d8"}
Oct 30 18:32:36.185653 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.185625    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"c8e9dc50b81932785b708e602096a4fdc5c2efa8d7a4cc1df86aca1bfda42500"}
Oct 30 18:32:36.188309 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.188285    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="d1fd27117c47bb926ad8fb7c97500e79177bb1a9f4d0e59762fb38f7f25d3ef2" exitCode=0
Oct 30 18:32:36.188403 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:36.188326    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"d1fd27117c47bb926ad8fb7c97500e79177bb1a9f4d0e59762fb38f7f25d3ef2"}
Oct 30 18:32:37.194040 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:37.193994    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="c15b14ad18eb9129b5893c5ba67686c26fac0cf0c66985f0eb594120c5f4f4fc" exitCode=0
Oct 30 18:32:37.194808 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:37.194110    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerDied","Data":"c15b14ad18eb9129b5893c5ba67686c26fac0cf0c66985f0eb594120c5f4f4fc"}
Oct 30 18:32:37.973573 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:37.973492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:37.973714 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:37.973577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:37.973714 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:37.973670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:37.973714 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:37.973684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:37.973861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:37.973746    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:37.973861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:37.973802    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:38.199651 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:38.199615    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:38.200068 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:38.200042    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"48df535d1c5f97f8249dd2afbb656af0e3eff6c776042002151e88c0c6d4acf4"}
Oct 30 18:32:38.203870 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:38.203842    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9pks" event={"ID":"9a74cbcd-a5c8-474b-8206-6f4b7e17b416","Type":"ContainerStarted","Data":"b6ebc4b1390c3ad02f8c67f2929a5f88c22453fe484ab638d85401e0eaa635ba"}
Oct 30 18:32:38.229443 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:38.229373    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9pks" podStartSLOduration=8.22877906 podStartE2EDuration="19.229361276s" totalImagesPullingTime="11.000582216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:38.226460694 +0000 UTC m=+19.693814047" watchObservedRunningTime="2026-10-30 18:32:38.229361276 +0000 UTC m=+19.696714627"
Oct 30 18:32:39.973328 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:39.973154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:39.974042 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:39.973154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:39.974042 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:39.973402    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:39.974042 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:39.973176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:39.974042 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:39.973467    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:39.974042 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:39.973565    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:40.211604 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.211511    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:40.211886 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.211864    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"54bb37c205106ab7ecdabab25d4ed5682eb6266ea709eb38978524c5be0928c5"}
Oct 30 18:32:40.212234 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.212210    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:40.212401 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.212382    2576 scope.go:122] "RemoveContainer" containerID="5b90f94b53a3c60b4a35d0b02adbd8310842582d2bcc75b9db4c1de0fc5571d8"
Oct 30 18:32:40.231248 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.231228    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:40.966099 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:40.966059    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:41.219635 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.219548    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:41.220053 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.220016    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" event={"ID":"eb49e957-8fde-4701-bbb0-b718a6acd947","Type":"ContainerStarted","Data":"506d165b6268913c460ecde5af9eecc07783533775e148a9fe511ce86bf2c41c"}
Oct 30 18:32:41.220471 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.220443    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:41.236469 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.236447    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:32:41.251839 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.251790    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k" podStartSLOduration=7.308002007 podStartE2EDuration="22.251773283s" totalImagesPullingTime="14.943771276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:41.250922389 +0000 UTC m=+22.718275744" watchObservedRunningTime="2026-10-30 18:32:41.251773283 +0000 UTC m=+22.719126639"
Oct 30 18:32:41.573040 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.572958    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w7prc"]
Oct 30 18:32:41.573278 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.573142    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:41.573278 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:41.573230    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:41.573559 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.573535    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5n4z"]
Oct 30 18:32:41.573671 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.573656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:41.573785 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:41.573763    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:41.575784 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.575766    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lg2zm"]
Oct 30 18:32:41.575856 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:41.575841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:41.575915 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:41.575902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:42.973337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:42.973302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:42.973337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:42.973320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:42.973817 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:42.973448    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:42.973817 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:42.973495    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:42.973817 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:42.973529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:42.973817 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:42.973629    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:44.973503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:44.973423    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:44.973503 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:44.973442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:44.974012 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:44.973560    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:44.974012 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:44.973606    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:44.974012 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:44.973665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:44.974012 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:44.973792    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:46.972983 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:46.972905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:46.972983 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:46.972929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:46.973552 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:46.973017    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lg2zm" podUID="afe4c4f3-230a-4545-844b-fd72139f8e6d"
Oct 30 18:32:46.973552 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:46.973058    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:46.973552 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:46.973188    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5n4z" podUID="db036c26-2a01-431b-a135-d3098bd05427"
Oct 30 18:32:46.973552 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:46.973264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7prc" podUID="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4"
Oct 30 18:32:47.221824 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:47.221793    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:47.221994 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:47.221899    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:32:47.222376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:47.222361    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:32:48.347169 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.346972    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeReady"
Oct 30 18:32:48.347552 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.347245    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:32:48.384474 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.384447    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vczpg"]
Oct 30 18:32:48.388676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.388646    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hghfk"]
Oct 30 18:32:48.389005 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.388931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:32:48.394050 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.393168    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-whf9r"]
Oct 30 18:32:48.394050 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.393372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:32:48.397553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.397531    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rsttd"]
Oct 30 18:32:48.397670 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.397651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:32:48.398012 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.397982    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.398139 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.398040    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.401232 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.401214    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"]
Oct 30 18:32:48.401356 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.401337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.405170 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.405146    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:32:48.406161 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.405276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:32:48.414079 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.414060    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:32:48.414796 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.414775    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56f55855fc-ctkk9"]
Oct 30 18:32:48.414937 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.414873    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lvtf9\""
Oct 30 18:32:48.416705 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.416687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.419137 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.419118    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"]
Oct 30 18:32:48.419388 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.419250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.423178 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.423157    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"]
Oct 30 18:32:48.423283 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.423257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.427373 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.427354    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"]
Oct 30 18:32:48.427583 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.427453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.431604 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.431583    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:32:48.431724 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.431709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.434713 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.434696    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"]
Oct 30 18:32:48.434842 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.434825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.437923 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.437900    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ghhlw"]
Oct 30 18:32:48.438105 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.438067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.441435 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.441414    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"]
Oct 30 18:32:48.441757 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.441697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.446759 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.446705    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"]
Oct 30 18:32:48.446881 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.446865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.451317 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451295    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vczpg"]
Oct 30 18:32:48.451432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451335    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-whf9r"]
Oct 30 18:32:48.451432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451357    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"]
Oct 30 18:32:48.451432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451376    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"]
Oct 30 18:32:48.451432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451398    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wrlwc"]
Oct 30 18:32:48.451432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.451409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:32:48.456244 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.456224    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l8bh7"]
Oct 30 18:32:48.456344 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.456326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.459740 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459717    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"]
Oct 30 18:32:48.459832 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459759    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:32:48.459832 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459785    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"]
Oct 30 18:32:48.459832 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459808    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"]
Oct 30 18:32:48.459832 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459829    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459854    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rsttd"]
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459875    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hghfk"]
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459896    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ghhlw"]
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459918    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrlwc"]
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459826    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.460029 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.459945    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56f55855fc-ctkk9"]
Oct 30 18:32:48.465333 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.465312    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"]
Oct 30 18:32:48.466611 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.466550    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"]
Oct 30 18:32:48.468142 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.468120    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8bh7"]
Oct 30 18:32:48.497412 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497388    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.497527 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497508    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.497601 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497552    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.497676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497663    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.497733 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497724    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.497799 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.497852 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497828    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:32:48.497985 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497970    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:32:48.498046 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497990    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:32:48.498046 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.497974    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.498046 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.498008    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.498046 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.498036    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.498232 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.498078    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.498232 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.498109    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.499419 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499397    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.499518 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499421    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499592 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499375    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:32:48.499654 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499641    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499709 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499657    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499772 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499741    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:32:48.499772 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499764    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499772    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499793    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499800    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499767    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499818    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:32:48.499884 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.499845    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:32:48.501346 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.501330    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:32:48.508708 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.508692    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:32:48.521568 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521551    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:32:48.521662 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521582    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:32:48.521752 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521738    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:32:48.521809 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521770    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:32:48.521809 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521779    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:32:48.521809 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521793    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:32:48.521809 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521805    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:32:48.521933 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.521814    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:32:48.523253 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523240    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:32:48.523712 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523700    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:32:48.523750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523707    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:32:48.523797 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523764    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:32:48.523797 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523787    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:32:48.523874 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523822    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pg8bd\""
Oct 30 18:32:48.523874 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523845    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:32:48.523874 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523869    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5wnwl\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523881    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523907    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523916    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2fw8l\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523925    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2hpwn\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523949    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g4tgh\""
Oct 30 18:32:48.523968 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523964    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n52cc\""
Oct 30 18:32:48.524186 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.523993    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:32:48.524186 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.524008    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6twlp\""
Oct 30 18:32:48.524186 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.524015    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7b9zd\""
Oct 30 18:32:48.524186 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.524019    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g8dfk\""
Oct 30 18:32:48.524186 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.524009    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j8xl4\""
Oct 30 18:32:48.525134 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.525114    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zs4rg\""
Oct 30 18:32:48.525235 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.525164    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-glmtq\""
Oct 30 18:32:48.527121 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.527205 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.527205 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl667\" (UniqueName: \"kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.527315 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.527315 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.527415 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh6r\" (UniqueName: \"kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r\") pod \"volume-data-source-validator-58c9f5f46d-ntj76\" (UID: \"9ce97a9d-4f14-469c-9258-12f174cf1ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:32:48.527415 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn78n\" (UniqueName: \"kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.527415 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.527557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.527557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.527557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.527557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxq7m\" (UniqueName: \"kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.527728 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5sl\" (UniqueName: \"kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25bl5\" (UniqueName: \"kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5\") pod \"kas-connection-checker-685895c7c9-whf9r\" (UID: \"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562\") " pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.527997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.528024 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhsn\" (UniqueName: \"kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24p4r\" (UniqueName: \"kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksl5\" (UniqueName: \"kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnrp\" (UniqueName: \"kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.528497 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zfx\" (UniqueName: \"kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkmw\" (UniqueName: \"kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw\") pod \"kas-connection-checker-685895c7c9-vczpg\" (UID: \"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4\") " pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfkt\" (UniqueName: \"kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt\") pod \"network-check-source-6f78d59f99-t9ncn\" (UID: \"156c7419-fd56-4fe4-aad9-b37b4e972a9f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpw6\" (UniqueName: \"kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6\") pod \"kas-connection-checker-685895c7c9-hghfk\" (UID: \"200b118b-598b-4885-a4d6-f570d78aa4a7\") " pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.528899 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.529337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.529337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.529337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.529337 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.528969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.629483 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.629483 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.629483 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.629669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.629699 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.629677    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:48.630260 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.630239    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls podName:263e2d58-6476-493e-9de0-319a8ed0c5f7 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.129716461 +0000 UTC m=+30.597069805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lf2ct" (UID: "263e2d58-6476-493e-9de0-319a8ed0c5f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:48.630432 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.630500 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.630557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.630711 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.630775 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.630874 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl667\" (UniqueName: \"kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.630949 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.630949 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.630949 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.631114 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.630959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh6r\" (UniqueName: \"kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r\") pod \"volume-data-source-validator-58c9f5f46d-ntj76\" (UID: \"9ce97a9d-4f14-469c-9258-12f174cf1ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:32:48.631114 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn78n\" (UniqueName: \"kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.631114 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.631114 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.631319 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.631319 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.631319 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.631466 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.631414    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.131397588 +0000 UTC m=+30.598750944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:32:48.631466 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.631831 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.631915 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.631996 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.631996 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.631980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.632104 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.632104 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxq7m\" (UniqueName: \"kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.632199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632123    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.632199 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.632294 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.632340 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5sl\" (UniqueName: \"kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.632388 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.632388 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.632481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.632481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.632576 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bl5\" (UniqueName: \"kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5\") pod \"kas-connection-checker-685895c7c9-whf9r\" (UID: \"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562\") " pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:32:48.632576 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.632576 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.632682    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.632700    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c4bf777f-scwmb: secret "image-registry-tls" not found
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhsn\" (UniqueName: \"kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.632753    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls podName:de844348-2c71-4773-b997-e1a79da8e4e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.132737553 +0000 UTC m=+30.600090900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls") pod "image-registry-59c4bf777f-scwmb" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1") : secret "image-registry-tls" not found
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.632896 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.632868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.633702 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.633450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.634167 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.634146    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:32:48.634256 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.634211    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.134195987 +0000 UTC m=+30.601549339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : secret "router-metrics-certs-default" not found
Oct 30 18:32:48.634321 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.634301    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:48.634321 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.634313    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b594796db-ldb2p: secret "image-registry-tls" not found
Oct 30 18:32:48.634457 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.634357    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls podName:4c39f772-3455-4174-a1b8-49b8a5f74b30 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.134344739 +0000 UTC m=+30.601698087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls") pod "image-registry-5b594796db-ldb2p" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30") : secret "image-registry-tls" not found
Oct 30 18:32:48.636999 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.634935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.636999 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.635217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.636999 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.636671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.637294 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.637273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.637946 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.637501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.637946 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.637541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24p4r\" (UniqueName: \"kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.637946 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.637573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.637946 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.637678    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:32:48.637946 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.637726    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls podName:8391c80c-4407-4fc4-9053-b336be68476c nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.137711715 +0000 UTC m=+30.605065051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls") pod "dns-default-l8bh7" (UID: "8391c80c-4407-4fc4-9053-b336be68476c") : secret "dns-default-metrics-tls" not found
Oct 30 18:32:48.638264 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksl5\" (UniqueName: \"kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.638264 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.638264 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnrp\" (UniqueName: \"kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.638264 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.638264 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.638481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.638481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zfx\" (UniqueName: \"kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.638481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.638481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkmw\" (UniqueName: \"kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw\") pod \"kas-connection-checker-685895c7c9-vczpg\" (UID: \"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4\") " pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:32:48.638481 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfkt\" (UniqueName: \"kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt\") pod \"network-check-source-6f78d59f99-t9ncn\" (UID: \"156c7419-fd56-4fe4-aad9-b37b4e972a9f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpw6\" (UniqueName: \"kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6\") pod \"kas-connection-checker-685895c7c9-hghfk\" (UID: \"200b118b-598b-4885-a4d6-f570d78aa4a7\") " pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:32:48.638704 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.638664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.639834 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.639810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.640201 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.640181    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:32:48.640293 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.640248    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.140230258 +0000 UTC m=+30.607583604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:32:48.640400 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.640376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.640908 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.640842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.640998 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.640970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.640998 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.640991    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.641178 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.641123    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:32:48.641178 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:48.641174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:32:49.141154245 +0000 UTC m=+30.608507581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:32:48.641295 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.641265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.641557 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.641514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.641815 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.641760    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.642524 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.642376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.642524 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.642432    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.643295 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.642662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.643295 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.642930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.643295 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.643288    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.644205 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.644186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.644395 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.644374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.644987 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.644962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.645131 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.645107    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.647465 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.647442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn78n\" (UniqueName: \"kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:48.647536 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.647467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:48.647711 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.647687    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh6r\" (UniqueName: \"kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r\") pod \"volume-data-source-validator-58c9f5f46d-ntj76\" (UID: \"9ce97a9d-4f14-469c-9258-12f174cf1ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:32:48.648254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.648229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl667\" (UniqueName: \"kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.648434 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.648412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhsn\" (UniqueName: \"kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:48.655536 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.655517    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.656926 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.656900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5sl\" (UniqueName: \"kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:48.657210 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.657194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkmw\" (UniqueName: \"kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw\") pod \"kas-connection-checker-685895c7c9-vczpg\" (UID: \"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4\") " pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:32:48.657620 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.657596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bl5\" (UniqueName: \"kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5\") pod \"kas-connection-checker-685895c7c9-whf9r\" (UID: \"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562\") " pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:32:48.660986 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.660968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfkt\" (UniqueName: \"kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt\") pod \"network-check-source-6f78d59f99-t9ncn\" (UID: \"156c7419-fd56-4fe4-aad9-b37b4e972a9f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:32:48.660986 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.660979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zfx\" (UniqueName: \"kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.661259 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.660966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24p4r\" (UniqueName: \"kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.662598 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.662554    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnrp\" (UniqueName: \"kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:48.662863 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.662844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxq7m\" (UniqueName: \"kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.663257 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.663231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksl5\" (UniqueName: \"kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.664212 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.664193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:48.677402 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.677382    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpw6\" (UniqueName: \"kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6\") pod \"kas-connection-checker-685895c7c9-hghfk\" (UID: \"200b118b-598b-4885-a4d6-f570d78aa4a7\") " pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:32:48.707188 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.707168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:32:48.718188 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.718168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:32:48.728779 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.728600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:32:48.735065 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.735042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:48.743636 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.743607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:32:48.787818 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.787693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:32:48.806491 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.801877    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:32:48.806491 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.806160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:32:48.819634 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.819573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:32:48.825676 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.825620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:32:48.974626 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.974605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:48.974768 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.974748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:48.974821 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.974810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:48.988320 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.988278    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:32:48.988320 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.988290    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\""
Oct 30 18:32:48.988473 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.988364    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\""
Oct 30 18:32:48.988473 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.988368    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:32:48.988473 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:48.988424    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\""
Oct 30 18:32:49.037705 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.037687    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-whf9r"]
Oct 30 18:32:49.040242 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:49.040214    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4b50cf_c2e9_43b7_b6e5_9fc9f3e16562.slice/crio-658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770 WatchSource:0}: Error finding container 658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770: Status 404 returned error can't find the container with id 658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770
Oct 30 18:32:49.050296 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.050249    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vczpg"]
Oct 30 18:32:49.052448 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.052426    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hghfk"]
Oct 30 18:32:49.054898 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.054602    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rsttd"]
Oct 30 18:32:49.058993 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:49.056032    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda927c7bb_ddf0_4f1d_aafd_34b7b3881fc4.slice/crio-061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263 WatchSource:0}: Error finding container 061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263: Status 404 returned error can't find the container with id 061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263
Oct 30 18:32:49.067988 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.067937    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"]
Oct 30 18:32:49.071582 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:49.071547    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce97a9d_4f14_469c_9258_12f174cf1ac4.slice/crio-f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e WatchSource:0}: Error finding container f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e: Status 404 returned error can't find the container with id f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.143850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.143960    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144007    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls podName:263e2d58-6476-493e-9de0-319a8ed0c5f7 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.143990724 +0000 UTC m=+31.611344070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lf2ct" (UID: "263e2d58-6476-493e-9de0-319a8ed0c5f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144182    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.144170159 +0000 UTC m=+31.611523695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144242    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144254    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c4bf777f-scwmb: secret "image-registry-tls" not found
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144289    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls podName:de844348-2c71-4773-b997-e1a79da8e4e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.144281608 +0000 UTC m=+31.611634952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls") pod "image-registry-59c4bf777f-scwmb" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1") : secret "image-registry-tls" not found
Oct 30 18:32:49.145427 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144339    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144376    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.144367462 +0000 UTC m=+31.611720812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : secret "router-metrics-certs-default" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144441    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144451    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b594796db-ldb2p: secret "image-registry-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144479    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls podName:4c39f772-3455-4174-a1b8-49b8a5f74b30 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.144469233 +0000 UTC m=+31.611822579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls") pod "image-registry-5b594796db-ldb2p" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30") : secret "image-registry-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144812    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144858    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls podName:8391c80c-4407-4fc4-9053-b336be68476c nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.14484847 +0000 UTC m=+31.612201802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls") pod "dns-default-l8bh7" (UID: "8391c80c-4407-4fc4-9053-b336be68476c") : secret "dns-default-metrics-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144916    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.144949    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.144938039 +0000 UTC m=+31.612291373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.145001    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:32:49.146856 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:49.145032    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.145020667 +0000 UTC m=+31.612374013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:32:49.154917 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.153919    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"]
Oct 30 18:32:49.154917 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.154034    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ghhlw"]
Oct 30 18:32:49.156888 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.156861    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"]
Oct 30 18:32:49.162378 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.161486    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"]
Oct 30 18:32:49.164911 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.164884    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrlwc"]
Oct 30 18:32:49.171167 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:49.169846    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb264fe31_7f47_4667_9563_3ec98cf45e9b.slice/crio-8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337 WatchSource:0}: Error finding container 8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337: Status 404 returned error can't find the container with id 8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337
Oct 30 18:32:49.236387 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.236365    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj" event={"ID":"822c5b89-fcb5-4102-bf57-9ca8e15b7c40","Type":"ContainerStarted","Data":"8e7e558a0055118003ec5aa4c296dd55cb18fbd22f7c880f3940ea1714ad5676"}
Oct 30 18:32:49.237810 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.237786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hghfk" event={"ID":"200b118b-598b-4885-a4d6-f570d78aa4a7","Type":"ContainerStarted","Data":"450cadeb03a29e4fa606d6f1d08d7b7bff92257237da9f5ff60508e4f8f8a6c6"}
Oct 30 18:32:49.237912 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.237831    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hghfk" event={"ID":"200b118b-598b-4885-a4d6-f570d78aa4a7","Type":"ContainerStarted","Data":"d0c060a2f4b4639a3ab29c312ca7e19bbf392d80ef23df00908aba2153dfeb9c"}
Oct 30 18:32:49.238976 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.238957    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" event={"ID":"0fab5545-eaa2-487e-890b-f21a689ee08c","Type":"ContainerStarted","Data":"4f192bcbce06187b99c864d272ee7593b20bde49b2777317608c9c09152f019b"}
Oct 30 18:32:49.240288 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.240265    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vczpg" event={"ID":"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4","Type":"ContainerStarted","Data":"5699b31c51329af2024707013d64def46d5313ace35060f2c08b819955e1367d"}
Oct 30 18:32:49.240385 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.240306    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vczpg" event={"ID":"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4","Type":"ContainerStarted","Data":"061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263"}
Oct 30 18:32:49.241281 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.241264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc" event={"ID":"8c676795-0942-4bfc-8abd-65ab1071d237","Type":"ContainerStarted","Data":"d5f9952e96c9bdc6ff2c020a3d524caa2b217df0e43345c793fbf24e17de41dd"}
Oct 30 18:32:49.242700 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.242679    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-whf9r" event={"ID":"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562","Type":"ContainerStarted","Data":"8cbc60284cd26fb0f65bfbad660ddec9258c9b7557a86c739fd3133e4109c30a"}
Oct 30 18:32:49.242797 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.242712    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-whf9r" event={"ID":"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562","Type":"ContainerStarted","Data":"658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770"}
Oct 30 18:32:49.244065 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.244029    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76" event={"ID":"9ce97a9d-4f14-469c-9258-12f174cf1ac4","Type":"ContainerStarted","Data":"f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e"}
Oct 30 18:32:49.245266 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.245243    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrlwc" event={"ID":"b264fe31-7f47-4667-9563-3ec98cf45e9b","Type":"ContainerStarted","Data":"8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337"}
Oct 30 18:32:49.246372 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.246352    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn" event={"ID":"156c7419-fd56-4fe4-aad9-b37b4e972a9f","Type":"ContainerStarted","Data":"639b7e27be45724ab6b25c02d9a38fb4ca12a0ef56e90ad29074f6de517e4e80"}
Oct 30 18:32:49.247398 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.247377    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ghhlw" event={"ID":"3b50a449-623e-44d0-bdb9-541e633dab54","Type":"ContainerStarted","Data":"4b7197c1bb05666968692759dfa6f9e0c395a29eff61f7acee2332369ccffbfa"}
Oct 30 18:32:49.260688 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:49.260652    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hghfk" podStartSLOduration=174.260640768 podStartE2EDuration="2m54.260640768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:49.259849703 +0000 UTC m=+30.727203057" watchObservedRunningTime="2026-10-30 18:32:49.260640768 +0000 UTC m=+30.727994113"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.151921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.151987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152127    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152101407 +0000 UTC m=+33.619454761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152198    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152212    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b594796db-ldb2p: secret "image-registry-tls" not found
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls podName:4c39f772-3455-4174-a1b8-49b8a5f74b30 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.15224506 +0000 UTC m=+33.619598402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls") pod "image-registry-5b594796db-ldb2p" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30") : secret "image-registry-tls" not found
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152269    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152308    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls podName:8391c80c-4407-4fc4-9053-b336be68476c nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152297633 +0000 UTC m=+33.619650971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls") pod "dns-default-l8bh7" (UID: "8391c80c-4407-4fc4-9053-b336be68476c") : secret "dns-default-metrics-tls" not found
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.152417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:50.152861 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152522    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls podName:263e2d58-6476-493e-9de0-319a8ed0c5f7 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152549424 +0000 UTC m=+33.619902770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lf2ct" (UID: "263e2d58-6476-493e-9de0-319a8ed0c5f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152617    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152626    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c4bf777f-scwmb: secret "image-registry-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152653    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls podName:de844348-2c71-4773-b997-e1a79da8e4e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152644348 +0000 UTC m=+33.619997695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls") pod "image-registry-59c4bf777f-scwmb" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1") : secret "image-registry-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152682    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152705    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152728    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152713979 +0000 UTC m=+33.620067334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152746    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.15273655 +0000 UTC m=+33.620089887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152774    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:32:50.153973 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:50.152810    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:52.152800881 +0000 UTC m=+33.620154248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : secret "router-metrics-certs-default" not found
Oct 30 18:32:50.280598 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.280544    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vczpg" podStartSLOduration=175.280523888 podStartE2EDuration="2m55.280523888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:50.278393769 +0000 UTC m=+31.745747124" watchObservedRunningTime="2026-10-30 18:32:50.280523888 +0000 UTC m=+31.747877244"
Oct 30 18:32:50.301717 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:50.301657    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-whf9r" podStartSLOduration=175.301640339 podStartE2EDuration="2m55.301640339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:50.30076638 +0000 UTC m=+31.768119729" watchObservedRunningTime="2026-10-30 18:32:50.301640339 +0000 UTC m=+31.768993694"
Oct 30 18:32:51.669467 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.669212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:32:51.670002 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:51.669464    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:32:51.670002 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:51.669578    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs podName:db036c26-2a01-431b-a135-d3098bd05427 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:23.669555022 +0000 UTC m=+65.136908373 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs") pod "network-metrics-daemon-c5n4z" (UID: "db036c26-2a01-431b-a135-d3098bd05427") : secret "metrics-daemon-secret" not found
Oct 30 18:32:51.670347 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.670224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:51.692418 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.684833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:51.701683 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.701659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:32:51.873371 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.873334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:51.879046 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.879017    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:51.989254 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:51.988929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:52.054302 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.054194    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lg2zm"]
Oct 30 18:32:52.062814 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:52.062779    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe4c4f3_230a_4545_844b_fd72139f8e6d.slice/crio-e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28 WatchSource:0}: Error finding container e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28: Status 404 returned error can't find the container with id e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.176503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.176635    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.176695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls podName:263e2d58-6476-493e-9de0-319a8ed0c5f7 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.176678248 +0000 UTC m=+37.644031597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lf2ct" (UID: "263e2d58-6476-493e-9de0-319a8ed0c5f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177116073 +0000 UTC m=+37.644469408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177213    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177224    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c4bf777f-scwmb: secret "image-registry-tls" not found
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177258    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls podName:de844348-2c71-4773-b997-e1a79da8e4e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177247752 +0000 UTC m=+37.644601087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls") pod "image-registry-59c4bf777f-scwmb" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1") : secret "image-registry-tls" not found
Oct 30 18:32:52.177750 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177332    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177352677 +0000 UTC m=+37.644706024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : secret "router-metrics-certs-default" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177419    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177427    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b594796db-ldb2p: secret "image-registry-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177455    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls podName:4c39f772-3455-4174-a1b8-49b8a5f74b30 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177445896 +0000 UTC m=+37.644799243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls") pod "image-registry-5b594796db-ldb2p" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30") : secret "image-registry-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177510    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177539    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls podName:8391c80c-4407-4fc4-9053-b336be68476c nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177529356 +0000 UTC m=+37.644882703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls") pod "dns-default-l8bh7" (UID: "8391c80c-4407-4fc4-9053-b336be68476c") : secret "dns-default-metrics-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177592    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177620    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177610638 +0000 UTC m=+37.644963976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177669    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:32:52.178649 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:52.177697    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.177688121 +0000 UTC m=+37.645041468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:32:52.266713 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.266625    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76" event={"ID":"9ce97a9d-4f14-469c-9258-12f174cf1ac4","Type":"ContainerStarted","Data":"68c84019db35709316a549b16b6a3d20e45f11093224e4d5d0c79b4e16b21872"}
Oct 30 18:32:52.274702 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.274673    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lg2zm" event={"ID":"afe4c4f3-230a-4545-844b-fd72139f8e6d","Type":"ContainerStarted","Data":"e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28"}
Oct 30 18:32:52.289504 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.289420    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76" podStartSLOduration=135.168323051 podStartE2EDuration="2m17.28940034s" totalImagesPullingTime="2.121077289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:52.286065626 +0000 UTC m=+33.753418981" watchObservedRunningTime="2026-10-30 18:32:52.28940034 +0000 UTC m=+33.756753694"
Oct 30 18:32:52.317691 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:52.317664    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w7prc"]
Oct 30 18:32:53.281232 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:53.281193    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w7prc" event={"ID":"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4","Type":"ContainerStarted","Data":"7cc5c7c8156b61efdd45adf3cd3ee5ed178a2b937217117dae191d0ba0a9161f"}
Oct 30 18:32:53.285513 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:53.285484    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/0.log"
Oct 30 18:32:53.285624 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:53.285526    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fab5545-eaa2-487e-890b-f21a689ee08c" containerID="c29f583f2bc7431459b0d899f14764856cc7b1ac16b6cf2ec870535155ef57b7" exitCode=255
Oct 30 18:32:53.286243 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:53.286224    2576 scope.go:122] "RemoveContainer" containerID="c29f583f2bc7431459b0d899f14764856cc7b1ac16b6cf2ec870535155ef57b7"
Oct 30 18:32:53.286479 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:53.286459    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" event={"ID":"0fab5545-eaa2-487e-890b-f21a689ee08c","Type":"ContainerDied","Data":"c29f583f2bc7431459b0d899f14764856cc7b1ac16b6cf2ec870535155ef57b7"}
Oct 30 18:32:54.292649 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.292369    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ghhlw" event={"ID":"3b50a449-623e-44d0-bdb9-541e633dab54","Type":"ContainerStarted","Data":"e18a8a8a133dd03cda366ecf6ddec4638b023e5f9b78a0ca59fec68dd1856414"}
Oct 30 18:32:54.294444 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.294423    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.294909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/0.log"
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.294935    2576 generic.go:356] "Generic (PLEG): container finished" podID="0fab5545-eaa2-487e-890b-f21a689ee08c" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492" exitCode=255
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.294988    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" event={"ID":"0fab5545-eaa2-487e-890b-f21a689ee08c","Type":"ContainerDied","Data":"5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492"}
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.295021    2576 scope.go:122] "RemoveContainer" containerID="c29f583f2bc7431459b0d899f14764856cc7b1ac16b6cf2ec870535155ef57b7"
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.295375    2576 scope.go:122] "RemoveContainer" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492"
Oct 30 18:32:54.295764 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:54.295564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rsttd_openshift-console-operator(0fab5545-eaa2-487e-890b-f21a689ee08c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" podUID="0fab5545-eaa2-487e-890b-f21a689ee08c"
Oct 30 18:32:54.298593 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.298376    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc" event={"ID":"8c676795-0942-4bfc-8abd-65ab1071d237","Type":"ContainerStarted","Data":"55001668c689b135b3a878a507c25b28b1e688bb8a90cdf2aed20b4ff8d093e5"}
Oct 30 18:32:54.315991 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.315781    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-ghhlw" podStartSLOduration=137.355318168 podStartE2EDuration="2m22.315765018s" totalImagesPullingTime="4.96044685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:54.31422958 +0000 UTC m=+35.781582935" watchObservedRunningTime="2026-10-30 18:32:54.315765018 +0000 UTC m=+35.783118375"
Oct 30 18:32:54.336080 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.335331    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc" podStartSLOduration=138.385737159 podStartE2EDuration="2m23.335315261s" totalImagesPullingTime="4.949578102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:54.334989093 +0000 UTC m=+35.802342448" watchObservedRunningTime="2026-10-30 18:32:54.335315261 +0000 UTC m=+35.802668617"
Oct 30 18:32:54.973126 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.973018    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"]
Oct 30 18:32:54.995694 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.995640    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"]
Oct 30 18:32:54.996068 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:54.996046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:32:55.008043 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.006995    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:55.008043 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.007199    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bmnq2\""
Oct 30 18:32:55.008226 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.008206    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:32:55.023032 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.022953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqd6\" (UniqueName: \"kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6\") pod \"migrator-66df5d496b-jdzz6\" (UID: \"c151045e-2a02-4d9e-824b-bb2d3a28902c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:32:55.024686 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.024657    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"]
Oct 30 18:32:55.025166 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.024860    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:32:55.039867 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.039845    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"]
Oct 30 18:32:55.064056 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.064034    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:32:55.094437 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.094415    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hvfzd"]
Oct 30 18:32:55.095188 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.094880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.115376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.114542    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:32:55.115376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.114580    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvfzd"]
Oct 30 18:32:55.115376 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.114691    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.122313 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.122143    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:32:55.122313 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.122239    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:32:55.122313 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.122289    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqrtm\""
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9kb\" (UniqueName: \"kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqd6\" (UniqueName: \"kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6\") pod \"migrator-66df5d496b-jdzz6\" (UID: \"c151045e-2a02-4d9e-824b-bb2d3a28902c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.123913 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.123862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652gv\" (UniqueName: \"kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv\") pod \"migrator-66df5d496b-2rn9w\" (UID: \"d83555bf-0a49-4c44-93c7-e341d040313e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:32:55.136942 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.136920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqd6\" (UniqueName: \"kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6\") pod \"migrator-66df5d496b-jdzz6\" (UID: \"c151045e-2a02-4d9e-824b-bb2d3a28902c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:32:55.229594 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.229517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.229726 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.229609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.229767 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.229757    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:32:55.229831 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.229820    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:55.729802905 +0000 UTC m=+37.197156237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : configmap "kube-rbac-proxy" not found
Oct 30 18:32:55.230127 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230201 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230201 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.230201 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.230366 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.230366 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652gv\" (UniqueName: \"kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv\") pod \"migrator-66df5d496b-2rn9w\" (UID: \"d83555bf-0a49-4c44-93c7-e341d040313e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:32:55.230366 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9kb\" (UniqueName: \"kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230474 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230506 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230765 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.230884 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.230868    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:55.230939 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.230927    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:55.730909906 +0000 UTC m=+37.198263251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:55.231631 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.230995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.231631 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.231301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.231631 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.231421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.245773 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.245748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.245940 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.245917    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9kb\" (UniqueName: \"kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.249406 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.249381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652gv\" (UniqueName: \"kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv\") pod \"migrator-66df5d496b-2rn9w\" (UID: \"d83555bf-0a49-4c44-93c7-e341d040313e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:32:55.305190 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.305152    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrlwc" event={"ID":"b264fe31-7f47-4667-9563-3ec98cf45e9b","Type":"ContainerStarted","Data":"519d3a47a2a8adaf397071076b93f837db2073381bc8af60d34626491e11aaf1"}
Oct 30 18:32:55.308071 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.308042    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj" event={"ID":"822c5b89-fcb5-4102-bf57-9ca8e15b7c40","Type":"ContainerStarted","Data":"bd1cd3697f1648efbe1008a98c276b0c3c066c82f8de035be0f3050b9cf62512"}
Oct 30 18:32:55.314725 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.314701    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:32:55.315434 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.315413    2576 scope.go:122] "RemoveContainer" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492"
Oct 30 18:32:55.315606 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.315583    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rsttd_openshift-console-operator(0fab5545-eaa2-487e-890b-f21a689ee08c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" podUID="0fab5545-eaa2-487e-890b-f21a689ee08c"
Oct 30 18:32:55.325584 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.325552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:32:55.356920 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.356870    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wrlwc" podStartSLOduration=1.4704572329999999 podStartE2EDuration="7.356853174s" totalImagesPullingTime="5.886395941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:55.33347219 +0000 UTC m=+36.800825544" watchObservedRunningTime="2026-10-30 18:32:55.356853174 +0000 UTC m=+36.824206529"
Oct 30 18:32:55.359957 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.358224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:32:55.377514 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.377464    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj" podStartSLOduration=141.006936164 podStartE2EDuration="2m26.377443141s" totalImagesPullingTime="5.370506977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:55.37642446 +0000 UTC m=+36.843777839" watchObservedRunningTime="2026-10-30 18:32:55.377443141 +0000 UTC m=+36.844796501"
Oct 30 18:32:55.447109 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.443193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:32:55.653692 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.653665    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"]
Oct 30 18:32:55.659100 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.659066    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jqr4n"]
Oct 30 18:32:55.679474 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.679448    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jqr4n"]
Oct 30 18:32:55.679574 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.679562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.686044 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.686022    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:32:55.687279 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.687258    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:32:55.687406 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.687305    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6hqgv\""
Oct 30 18:32:55.687406 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.687337    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:32:55.687521 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.687465    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:32:55.699133 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.698726    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"]
Oct 30 18:32:55.703367 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:55.703344    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83555bf_0a49_4c44_93c7_e341d040313e.slice/crio-24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8 WatchSource:0}: Error finding container 24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8: Status 404 returned error can't find the container with id 24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.739604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bhv\" (UniqueName: \"kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.739719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.739786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.739815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.739880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.739959    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.740022    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.740004212 +0000 UTC m=+38.207357550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : configmap "kube-rbac-proxy" not found
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.740432    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:55.740553 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:55.740479    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:56.740468091 +0000 UTC m=+38.207821435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:55.772711 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.772680    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:32:55.840635 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.840600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bhv\" (UniqueName: \"kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.840748 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.840699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.840815 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.840753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.841537 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.841513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.843933 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.843891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.854985 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:55.854964    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bhv\" (UniqueName: \"kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:55.943039 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:55.943005    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5210ed_818c_48af_9883_0919354e4ae3.slice/crio-44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd WatchSource:0}: Error finding container 44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd: Status 404 returned error can't find the container with id 44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd
Oct 30 18:32:56.017257 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.017233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.244964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.245253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245392    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245405    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c4bf777f-scwmb: secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245457    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls podName:de844348-2c71-4773-b997-e1a79da8e4e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.245437539 +0000 UTC m=+45.712790871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls") pod "image-registry-59c4bf777f-scwmb" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1") : secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245798    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245836    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.245824293 +0000 UTC m=+45.713177625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : secret "router-metrics-certs-default" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245896    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245904    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b594796db-ldb2p: secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245927    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls podName:4c39f772-3455-4174-a1b8-49b8a5f74b30 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.245919936 +0000 UTC m=+45.713273268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls") pod "image-registry-5b594796db-ldb2p" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30") : secret "image-registry-tls" not found
Oct 30 18:32:56.246322 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245977    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.245998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls podName:8391c80c-4407-4fc4-9053-b336be68476c nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.245992178 +0000 UTC m=+45.713345510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls") pod "dns-default-l8bh7" (UID: "8391c80c-4407-4fc4-9053-b336be68476c") : secret "dns-default-metrics-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246038    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246060    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.246052573 +0000 UTC m=+45.713405906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246133    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246156    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.246149373 +0000 UTC m=+45.713502705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246190    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246210    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls podName:263e2d58-6476-493e-9de0-319a8ed0c5f7 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.246203921 +0000 UTC m=+45.713557252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lf2ct" (UID: "263e2d58-6476-493e-9de0-319a8ed0c5f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:32:56.246909 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.246253    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.246247214 +0000 UTC m=+45.713600546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:32:56.321157 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.321137    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-glkck-6hcwm" event={"ID":"6a5210ed-818c-48af-9883-0919354e4ae3","Type":"ContainerStarted","Data":"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"}
Oct 30 18:32:56.321518 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.321201    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-glkck-6hcwm" event={"ID":"6a5210ed-818c-48af-9883-0919354e4ae3","Type":"ContainerStarted","Data":"44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd"}
Oct 30 18:32:56.322524 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.322499    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6" event={"ID":"c151045e-2a02-4d9e-824b-bb2d3a28902c","Type":"ContainerStarted","Data":"7b5c18b284a8ac73f81666bb36c63e13125dc6a05aa648652c9b3026cf77c7af"}
Oct 30 18:32:56.324282 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.324259    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn" event={"ID":"156c7419-fd56-4fe4-aad9-b37b4e972a9f","Type":"ContainerStarted","Data":"fc2ccc773c7f2a82ff3501586eacf8074fcebb3625cf2d2703446e38d40629f1"}
Oct 30 18:32:56.326317 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.326276    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w" event={"ID":"d83555bf-0a49-4c44-93c7-e341d040313e","Type":"ContainerStarted","Data":"24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8"}
Oct 30 18:32:56.330424 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.330402    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jqr4n"]
Oct 30 18:32:56.333931 ip-10-0-131-155 kubenswrapper[2576]: W1030 18:32:56.333909    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a5d43a_1a55_4e76_a947_629f8fe5e76e.slice/crio-c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31 WatchSource:0}: Error finding container c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31: Status 404 returned error can't find the container with id c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31
Oct 30 18:32:56.347790 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.347743    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-glkck-6hcwm" podStartSLOduration=1.347728844 podStartE2EDuration="1.347728844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:56.346440156 +0000 UTC m=+37.813793513" watchObservedRunningTime="2026-10-30 18:32:56.347728844 +0000 UTC m=+37.815082199"
Oct 30 18:32:56.369113 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.368999    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn" podStartSLOduration=134.541096379 podStartE2EDuration="2m21.368982971s" totalImagesPullingTime="6.827886592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:56.36813756 +0000 UTC m=+37.835490919" watchObservedRunningTime="2026-10-30 18:32:56.368982971 +0000 UTC m=+37.836336326"
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.750908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:56.751025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.751247    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.751312    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:58.751294696 +0000 UTC m=+40.218648042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : configmap "kube-rbac-proxy" not found
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.751580    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:56.751674 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:56.751642    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:58.751626552 +0000 UTC m=+40.218979910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:57.334865 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:57.334821    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w7prc" event={"ID":"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4","Type":"ContainerStarted","Data":"2123c852b7600200fc39e5b1898af2501a3388fd5b80472543311a344e77d8ed"}
Oct 30 18:32:57.335674 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:57.335653    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:32:57.338678 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:57.338653    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n" event={"ID":"67a5d43a-1a55-4e76-a947-629f8fe5e76e","Type":"ContainerStarted","Data":"d4ef8b17ba61804b4ab7724048d53b9d32c834b7a45ab0e2b904bb1109856664"}
Oct 30 18:32:57.338753 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:57.338700    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n" event={"ID":"67a5d43a-1a55-4e76-a947-629f8fe5e76e","Type":"ContainerStarted","Data":"c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31"}
Oct 30 18:32:57.369044 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:57.368994    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w7prc" podStartSLOduration=34.398456175 podStartE2EDuration="38.36897799s" totalImagesPullingTime="3.970521815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:57.367965139 +0000 UTC m=+38.835318496" watchObservedRunningTime="2026-10-30 18:32:57.36897799 +0000 UTC m=+38.836331348"
Oct 30 18:32:58.343781 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.343690    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6" event={"ID":"c151045e-2a02-4d9e-824b-bb2d3a28902c","Type":"ContainerStarted","Data":"27d894b4dd4f43375cbbc0a39658376145207fc316f83f2df89ec43d63b6dc84"}
Oct 30 18:32:58.345495 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.345460    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w" event={"ID":"d83555bf-0a49-4c44-93c7-e341d040313e","Type":"ContainerStarted","Data":"fd361e7e5bf56d0f35dd2d92d3ac97979bd1df677ba60cd9bc0486920b3802e5"}
Oct 30 18:32:58.365651 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.365599    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n" podStartSLOduration=3.365583877 podStartE2EDuration="3.365583877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:57.39647642 +0000 UTC m=+38.863829774" watchObservedRunningTime="2026-10-30 18:32:58.365583877 +0000 UTC m=+39.832937234"
Oct 30 18:32:58.366288 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.366237    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6" podStartSLOduration=2.367151084 podStartE2EDuration="4.366222688s" totalImagesPullingTime="1.999071604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:58.36257773 +0000 UTC m=+39.829931084" watchObservedRunningTime="2026-10-30 18:32:58.366222688 +0000 UTC m=+39.833576044"
Oct 30 18:32:58.386822 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.386780    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w" podStartSLOduration=2.205970995 podStartE2EDuration="4.386764753s" totalImagesPullingTime="2.180793758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:58.384859878 +0000 UTC m=+39.852213231" watchObservedRunningTime="2026-10-30 18:32:58.386764753 +0000 UTC m=+39.854118113"
Oct 30 18:32:58.617151 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.617067    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-prl4g_0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/dns-node-resolver/0.log"
Oct 30 18:32:58.735342 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.735307    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:58.735499 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.735411    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:32:58.735808 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.735793    2576 scope.go:122] "RemoveContainer" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492"
Oct 30 18:32:58.736030 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:58.736003    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rsttd_openshift-console-operator(0fab5545-eaa2-487e-890b-f21a689ee08c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" podUID="0fab5545-eaa2-487e-890b-f21a689ee08c"
Oct 30 18:32:58.772995 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.772702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:58.772995 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.772895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:58.773212 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:58.773080    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:58.773212 ip-10-0-131-155 kubenswrapper[2576]: E1030 18:32:58.773170    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:02.773154945 +0000 UTC m=+44.240508280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:32:58.773421 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:58.773399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:32:59.416232 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:59.416214    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fqpv4_2fa09131-1c47-4d63-82d2-c5174c60fd49/node-ca/0.log"
Oct 30 18:32:59.671435 ip-10-0-131-155 kubenswrapper[2576]: I1030 18:32:59.671328    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:59.671498 ip-10-0-131-155 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:32:59.680507 ip-10-0-131-155 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:32:59.680682 ip-10-0-131-155 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:32:59.680732 ip-10-0-131-155 systemd[1]: kubelet.service: Consumed 3.115s CPU time.
Oct 30 18:32:59.690665 ip-10-0-131-155 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:32:59.797601 ip-10-0-131-155 kubenswrapper[5065]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:59.797601 ip-10-0-131-155 kubenswrapper[5065]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:32:59.797601 ip-10-0-131-155 kubenswrapper[5065]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:59.797953 ip-10-0-131-155 kubenswrapper[5065]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:59.800955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800882    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.800955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800953    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.800955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800957    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.800955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800960    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800963    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800966    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800969    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800971    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800974    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800976    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800979    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800981    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800984    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800987    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800990    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800993    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.800996    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801000    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801004    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801008    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801012    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801016    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801019    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.801148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801021    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801024    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801027    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801029    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801032    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801035    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801038    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801041    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801043    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801046    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801049    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801051    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801054    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801057    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801060    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801062    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801065    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801067    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801070    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.801643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801074    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801076    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801078    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801103    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801106    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801109    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801112    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801115    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801125    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801128    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801130    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801133    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801139    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801143    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801147    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801150    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801152    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801155    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801157    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.802190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801160    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801163    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801165    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801168    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801171    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801173    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801177    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801181    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801184    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801187    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801189    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801192    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801194    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801197    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801199    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801202    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801205    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801208    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801210    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801213    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.802661 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801216    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801218    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801226    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801229    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801231    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801234    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801236    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801239    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801242    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801244    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801247    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801249    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801252    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801254    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801257    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801260    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801263    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801266    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801268    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801271    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.803168 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801274    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801276    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801279    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801282    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.801285    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802205    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802213    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802216    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802219    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802222    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802224    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802227    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802229    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802232    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802234    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802238    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802241    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802244    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802246    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802249    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.803716 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802252    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802255    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802257    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802260    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802262    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802265    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802267    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802270    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802273    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802276    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802278    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802281    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802283    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802286    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802288    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802291    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802293    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802296    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802298    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802301    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.804247 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802303    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802305    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802308    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802310    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802313    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802316    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802319    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802321    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802324    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802327    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802329    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802331    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802334    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802337    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802339    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802342    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802344    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802347    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802349    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.804727 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802354    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802359    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802363    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802365    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802368    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802371    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802374    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802376    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802379    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802381    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802384    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802386    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802389    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802391    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802394    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802396    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802399    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802401    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802404    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802406    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.805219 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802409    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802411    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802414    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802417    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802419    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802422    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802424    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802427    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802430    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802432    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802435    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802440    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802444    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802447    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802449    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802452    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802454    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802457    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802459    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.805728 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802462    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802464    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802467    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802469    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802472    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802474    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802477    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802479    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802482    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802484    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802486    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802489    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802491    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802951    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802955    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802959    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802961    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802964    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802967    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802970    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.806201 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802973    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802975    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802978    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802981    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802984    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802986    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802989    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802992    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802994    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802997    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.802999    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803002    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803004    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803007    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803010    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803012    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803014    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803017    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803019    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.806725 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803022    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803024    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803027    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803029    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803032    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803035    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803037    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803040    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803043    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803045    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803048    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803050    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803053    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803056    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803058    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803061    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803063    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803066    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803068    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.807231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803070    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803073    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803075    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803078    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803097    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803101    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803104    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803106    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803109    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803112    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803114    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803117    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803120    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803122    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803125    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803127    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803130    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803132    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803135    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803138    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.807700 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803140    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803143    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803145    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803149    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803151    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803154    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803156    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803159    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803164    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803167    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803170    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803173    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803175    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803178    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803181    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803183    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803185    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803188    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803190    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.808235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803193    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803196    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803198    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803200    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803203    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803205    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803208    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803210    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803213    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803215    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803219    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803222    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803228    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803230    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803233    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803235    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803237    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803240    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803243    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803245    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.808737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803248    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.803252    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803350    5065 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803369    5065 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803378    5065 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803384    5065 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803389    5065 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803392    5065 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803396    5065 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803400    5065 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803403    5065 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803406    5065 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803409    5065 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803418    5065 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803421    5065 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803424    5065 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803426    5065 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803430    5065 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803433    5065 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803436    5065 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803440    5065 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803442    5065 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803445    5065 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803448    5065 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:32:59.809257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803453    5065 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803461    5065 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803464    5065 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803467    5065 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803470    5065 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803473    5065 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803476    5065 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803479    5065 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803482    5065 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803485    5065 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803489    5065 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803492    5065 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803495    5065 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803498    5065 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803501    5065 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803504    5065 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803508    5065 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803511    5065 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803514    5065 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803517    5065 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803520    5065 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803524    5065 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803527    5065 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803530    5065 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803533    5065 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:32:59.809833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803537    5065 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803540    5065 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803543    5065 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803546    5065 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803549    5065 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803551    5065 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803554    5065 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803559    5065 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803562    5065 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803566    5065 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803569    5065 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803572    5065 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803575    5065 flags.go:64] FLAG: --help="false"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803577    5065 flags.go:64] FLAG: --hostname-override="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803581    5065 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803584    5065 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803587    5065 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803590    5065 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803593    5065 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803596    5065 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803599    5065 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803602    5065 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803604    5065 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:32:59.810469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803607    5065 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803610    5065 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803622    5065 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803625    5065 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803628    5065 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803631    5065 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803634    5065 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803637    5065 flags.go:64] FLAG: --lock-file=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803640    5065 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803644    5065 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803646    5065 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803652    5065 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803655    5065 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803658    5065 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803661    5065 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803664    5065 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803668    5065 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803671    5065 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803674    5065 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803678    5065 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803681    5065 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803685    5065 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803688    5065 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803691    5065 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803694    5065 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:32:59.811130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803697    5065 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803700    5065 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803703    5065 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803706    5065 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803713    5065 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803716    5065 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803719    5065 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803722    5065 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803725    5065 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803729    5065 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803732    5065 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803734    5065 flags.go:64] FLAG: --port="10250"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803737    5065 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803740    5065 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-064129740e07dcfea"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803743    5065 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803746    5065 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803750    5065 flags.go:64] FLAG: --register-node="true"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803753    5065 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803757    5065 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803760    5065 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803763    5065 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803766    5065 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803770    5065 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803773    5065 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803776    5065 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803779    5065 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:32:59.811779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803782    5065 flags.go:64] FLAG: --runonce="false"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803785    5065 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803788    5065 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803791    5065 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803793    5065 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803796    5065 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803799    5065 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803802    5065 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803805    5065 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803808    5065 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803811    5065 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803814    5065 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803816    5065 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803819    5065 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803822    5065 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803825    5065 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803830    5065 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803833    5065 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803835    5065 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803839    5065 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803842    5065 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803844    5065 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803848    5065 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803851    5065 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803854    5065 flags.go:64] FLAG: --v="2"
Oct 30 18:32:59.812606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803858    5065 flags.go:64] FLAG: --version="false"
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803862    5065 flags.go:64] FLAG: --vmodule=""
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803866    5065 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.803869    5065 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804001    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804004    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804008    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804010    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804013    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804016    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804018    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804021    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804023    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804026    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804029    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804031    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804034    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804036    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804039    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804041    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804044    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.813629 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804046    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804049    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804052    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804054    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804057    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804059    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804062    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804064    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804068    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804070    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804073    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804075    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804078    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804097    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804102    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804106    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804110    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804114    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804117    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.814328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804122    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804125    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804128    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804131    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804134    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804136    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804139    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804142    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804145    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804147    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804150    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804152    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804155    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804158    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804160    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804163    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804166    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804168    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804170    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.814846 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804173    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804175    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804179    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804182    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804184    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804187    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804190    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804192    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804195    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804197    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804200    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804202    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804205    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804207    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804210    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804212    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804215    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804217    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804219    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804222    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.815367 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804224    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804228    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804230    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804233    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804237    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804239    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804242    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804244    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804248    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804252    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804256    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804259    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804262    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804264    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804269    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804271    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804274    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804276    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804279    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.815854 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804281    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804284    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804286    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804289    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804291    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804294    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804296    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804299    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804301    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804304    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804306    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804308    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.804336    5065 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804370    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804374    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804378    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.816363 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804381    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804386    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804388    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804391    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804393    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804396    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804399    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804401    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804404    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804407    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804409    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804412    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804416    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804419    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804421    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804424    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804426    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804429    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804431    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804433    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.817068 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804436    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804438    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804441    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804443    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804446    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804448    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804451    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804453    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804455    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804459    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804463    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804466    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804470    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804474    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804476    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804479    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804481    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804484    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804487    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.817750 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804490    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804492    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804494    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804497    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804499    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804502    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804505    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804508    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804510    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804513    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804515    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804518    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804520    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804523    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804525    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804528    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804530    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804532    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804535    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804537    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.818240 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804540    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804542    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804545    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804547    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804549    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804552    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804556    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804558    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804561    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804563    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804566    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804568    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804570    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804573    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804575    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804578    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804580    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804583    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804586    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804588    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.818719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804591    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804593    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804596    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804598    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804601    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804603    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804606    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804608    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804611    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804615    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804618    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804621    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804624    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804626    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804629    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804631    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804634    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804636    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804640    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.819220 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804646    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804649    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804651    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804654    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.804656    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.808319    5065 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.808338    5065 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.808357    5065 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.808361    5065 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808419    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808425    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808428    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808432    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808434    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808437    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808440    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808442    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808445    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808448    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.819696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808450    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808453    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808455    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808457    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808460    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808462    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808466    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808469    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808472    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808475    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808478    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808480    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808484    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808489    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808494    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808498    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808501    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808503    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808506    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808509    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.820294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808512    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808515    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808517    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808520    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808522    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808524    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808527    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808529    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808532    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808534    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808536    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808539    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808542    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808544    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808546    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808549    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808551    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808554    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808556    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808559    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.820989 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808562    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808566    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808571    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808576    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808579    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808581    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808584    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808586    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808589    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808591    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808594    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808597    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808600    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808603    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808607    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808611    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808614    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808616    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808619    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.821744 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808622    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808625    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808627    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808629    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808632    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808634    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808637    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808640    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808645    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808649    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808654    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808657    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808660    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808662    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808665    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808668    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808670    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808673    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808676    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.822421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808678    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808680    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808683    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808686    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808688    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808691    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808693    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808696    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808699    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808702    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808704    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808707    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808709    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808711    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808714    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808717    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808722    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.822875 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808726    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.808757    5065 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808900    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808906    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808910    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808912    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808916    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808920    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808923    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808925    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808927    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808930    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808932    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808935    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808937    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808940    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.823315 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808942    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808945    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808947    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808949    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808953    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808957    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808962    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808967    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808970    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808972    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808974    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808978    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808982    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808985    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808988    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808991    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808993    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808996    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.808998    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.823719 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809000    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809003    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809005    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809008    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809010    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809012    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809015    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809017    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809020    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809022    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809025    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809027    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809030    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809034    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809038    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809043    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809045    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809048    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809050    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.824264 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809053    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809055    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809058    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809063    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809066    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809068    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809071    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809073    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809076    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809078    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809081    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809100    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809104    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809107    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809110    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809112    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809115    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809117    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809119    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.824737 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809122    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809124    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809127    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809130    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809132    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809135    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809138    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809140    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809142    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809145    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809147    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809150    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809152    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809154    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809157    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809159    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809164    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809166    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809169    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.825215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809171    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809174    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809176    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809179    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809181    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809183    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809186    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809188    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809191    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809193    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809195    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809198    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809200    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809203    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809205    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809207    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.825715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.809231    5065 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809262    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809267    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809269    5065 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809272    5065 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809275    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809277    5065 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809280    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809282    5065 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809285    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809287    5065 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809289    5065 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809292    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809295    5065 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809298    5065 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809300    5065 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809303    5065 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809306    5065 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809308    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:59.826231 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809311    5065 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809313    5065 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809316    5065 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809318    5065 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809320    5065 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809323    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809325    5065 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809328    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809330    5065 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809333    5065 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809335    5065 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809337    5065 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809340    5065 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809342    5065 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809345    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809347    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809349    5065 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809352    5065 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809354    5065 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809358    5065 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:59.826682 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809362    5065 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809365    5065 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809367    5065 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809370    5065 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809372    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809374    5065 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809377    5065 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809380    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809383    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809385    5065 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809388    5065 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809390    5065 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809393    5065 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809395    5065 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809397    5065 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809400    5065 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809402    5065 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809405    5065 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809407    5065 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:59.827349 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809409    5065 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809413    5065 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809417    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809420    5065 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809422    5065 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809425    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809427    5065 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809429    5065 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809432    5065 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809434    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809436    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809439    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809441    5065 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809443    5065 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809446    5065 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809448    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809451    5065 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809453    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809456    5065 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:59.827912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809458    5065 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809460    5065 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809463    5065 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809466    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809469    5065 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809471    5065 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809473    5065 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809476    5065 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809478    5065 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809481    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809483    5065 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809485    5065 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809488    5065 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809490    5065 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809492    5065 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809495    5065 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809497    5065 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809499    5065 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809502    5065 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:59.828479 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809504    5065 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809507    5065 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809509    5065 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809511    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809514    5065 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809516    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809519    5065 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809521    5065 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809524    5065 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809526    5065 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:32:59.809529    5065 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.809577    5065 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810481    5065 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810537    5065 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810721    5065 server.go:1014] "Starting client certificate rotation"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810843    5065 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810883    5065 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:18 +0000 UTC" deadline="2028-06-16 20:39:11.184882463 +0000 UTC"
Oct 30 18:32:59.828990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.810909    5065 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14282h6m11.3739747s"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.811143    5065 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.812168    5065 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.814597    5065 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.816298    5065 log.go:25] "Validated CRI v1 image API"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.817129    5065 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:32:59.829538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.819006    5065 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7d3c7a3-4135-4ad8-8d72-d2a259ab25b9:/dev/nvme0n1p3 f6e0f408-4753-4d4f-8d02-45ad59897843:/dev/nvme0n1p4]
Oct 30 18:32:59.829864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.819026    5065 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b7197c1bb05666968692759dfa6f9e0c395a29eff61f7acee2332369ccffbfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b7197c1bb05666968692759dfa6f9e0c395a29eff61f7acee2332369ccffbfa/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f192bcbce06187b99c864d272ee7593b20bde49b2777317608c9c09152f019b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f192bcbce06187b99c864d272ee7593b20bde49b2777317608c9c09152f019b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/639b7e27be45724ab6b25c02d9a38fb4ca12a0ef56e90ad29074f6de517e4e80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/639b7e27be45724ab6b25c02d9a38fb4ca12a0ef56e90ad29074f6de517e4e80/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b5c18b284a8ac73f81666bb36c63e13125dc6a05aa648652c9b3026cf77c7af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b5c18b284a8ac73f81666bb36c63e13125dc6a05aa648652c9b3026cf77c7af/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cc5c7c8156b61efdd45adf3cd3ee5ed178a2b937217117dae191d0ba0a9161f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cc5c7c8156b61efdd45adf3cd3ee5ed178a2b937217117dae191d0ba0a9161f/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e7e558a0055118003ec5aa4c296dd55cb18fbd22f7c880f3940ea1714ad5676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e7e558a0055118003ec5aa4c296dd55cb18fbd22f7c880f3940ea1714ad5676/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4/userdata/shm major:0 minor:59 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0c060a2f4b4639a3ab29c312ca7e19bbf392d80ef23df00908aba2153dfeb9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0c060a2f4b4639a3ab29c312ca7e19bbf392d80ef23df00908aba2153dfeb9c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5f9952e96c9bdc6ff2c020a3d524caa2b217df0e43345c793fbf24e17de41dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5f9952e96c9bdc6ff2c020a3d524caa2b217df0e43345c793fbf24e17de41dd/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/volumes/kubernetes.io~projected/kube-api-access-pt6pz:{mountpoint:/var/lib/kubelet/pods/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/volumes/kubernetes.io~projected/kube-api-access-pt6pz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~projected/kube-api-access-kl667:{mountpoint:/var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~projected/kube-api-access-kl667 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/156c7419-fd56-4fe4-aad9-b37b4e972a9f/volumes/kubernetes.io~projected/kube-api-access-ltfkt:{mountpoint:/var/lib/kubelet/pods/156c7419-fd56-4fe4-aad9-b37b4e972a9f/volumes/kubernetes.io~projected/kube-api-access-ltfkt major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4/volumes/kubernetes.io~projected/kube-api-access-42v47:{mountpoint:/var/lib/kubelet/pods/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4/volumes/kubernetes.io~projected/kube-api-access-42v47 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/200b118b-598b-4885-a4d6-f570d78aa4a7/volumes/kubernetes.io~projected/kube-api-access-zwpw6:{mountpoint:/var/lib/kubelet/pods/200b118b-598b-4885-a4d6-f570d78aa4a7/volumes/kubernetes.io~projected/kube-api-access-zwpw6 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/263e2d58-6476-493e-9de0-319a8ed0c5f7/volumes/kubernetes.io~projected/kube-api-access-rdnrp:{mountpoint:/var/lib/kubelet/pods/263e2d58-6476-493e-9de0-319a8ed0c5f7/volumes/kubernetes.io~projected/kube-api-access-rdnrp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fa09131-1c47-4d63-82d2-c5174c60fd49/volumes/kubernetes.io~projected/kube-api-access-l2rgf:{mountpoint:/var/lib/kubelet/pods/2fa09131-1c47-4d63-82d2-c5174c60fd49/volumes/kubernetes.io~projected/kube-api-access-l2rgf major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3392f519-fd88-4b83-bda6-f7489c047423/volumes/kubernetes.io~projected/kube-api-access-j7hkd:{mountpoint:/var/lib/kubelet/pods/3392f519-fd88-4b83-bda6-f7489c047423/volumes/kubernetes.io~projected/kube-api-access-j7hkd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~projected/kube-api-access-zksl5:{mountpoint:/var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~projected/kube-api-access-zksl5 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~projected/kube-api-access-xn78n:{mountpoint:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~projected/kube-api-access-xn78n major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562/volumes/kubernetes.io~projected/kube-api-access-25bl5:{mountpoint:/var/lib/kubelet/pods/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562/volumes/kubernetes.io~projected/kube-api-access-25bl5 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fb2a473-98fc-4918-a07e-fb7d4edfddb8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3fb2a473-98fc-4918-a07e-fb7d4edfddb8/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/kube-api-access-hwpcn:{mountpoint:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/kube-api-access-hwpcn major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~projected/kube-api-access-c8bhv:{mountpoint:/var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~projected/kube-api-access-c8bhv major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~secret/signing-key major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a5210ed-818c-48af-9883-0919354e4ae3/volumes/kubernetes.io~projected/kube-api-access-npxd6:{mountpoint:/var/lib/kubelet/pods/6a5210ed-818c-48af-9883-0919354e4ae3/volumes/kubernetes.io~projected/kube-api-access-npxd6 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~projected/kube-api-access-bxq7m:{mountpoint:/var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~projected/kube-api-access-bxq7m major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8391c80c-4407-4fc4-9053-b336be68476c/volumes/kubernetes.io~projected/kube-api-access-cx5sl:{mountpoint:/var/lib/kubelet/pods/8391c80c-4407-4fc4-9053-b336be68476c/volumes/kubernetes.io~projected/kube-api-access-cx5sl major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~projected/kube-api-access-84zfx:{mountpoint:/var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~projected/kube-api-access-84zfx major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a74cbcd-a5c8-474b-8206-6f4b7e17b416/volumes/kubernetes.io~projected/kube-api-access-fncxb:{mountpoint:/var/lib/kubelet/pods/9a74cbcd-a5c8-474b-8206-6f4b7e17b416/volumes/kubernetes.io~projected/kube-api-access-fncxb major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ce97a9d-4f14-469c-9258-12f174cf1ac4/volumes/kubernetes.io~projected/kube-api-access-2zh6r:{mountpoint:/var/lib/kubelet/pods/9ce97a9d-4f14-469c-9258-12f174cf1ac4/volumes/kubernetes.io~projected/kube-api-access-2zh6r major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~projected/kube-api-access-6wbvm:{mountpoint:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~projected/kube-api-access-6wbvm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4/volumes/kubernetes.io~projected/kube-api-access-nqkmw:{mountpoint:/var/lib/kubelet/pods/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4/volumes/kubernetes.io~projected/kube-api-access-nqkmw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afe4c4f3-230a-4545-844b-fd72139f8e6d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/afe4c4f3-230a-4545-844b-fd72139f8e6d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~projected/kube-api-access-24p4r:{mountpoint:/var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~projected/kube-api-access-24p4r major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c151045e-2a02-4d9e-824b-bb2d3a28902c/volumes/kubernetes.io~projected/kube-api-access-wjqd6:{mountpoint:/var/lib/kubelet/pods/c151045e-2a02-4d9e-824b-bb2d3a28902c/volumes/kubernetes.io~projected/kube-api-access-wjqd6 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d741136f-7a0a-479a-84ea-d74bdc485189/volumes/kubernetes.io~projected/kube-api-access-zgr5m:{mountpoint:/var/lib/kubelet/pods/d741136f-7a0a-479a-84ea-d74bdc485189/volumes/kubernetes.io~projected/kube-api-access-zgr5m major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83555bf-0a49-4c44-93c7-e341d040313e/volumes/kubernetes.io~projected/kube-api-access-652gv:{mountpoint:/var/lib/kubelet/pods/d83555bf-0a49-4c44-93c7-e341d040313e/volumes/kubernetes.io~projected/kube-api-access-652gv major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d961e0b7-299c-413d-8206-d7e9398e76b2/volumes/kubernetes.io~projected/kube-api-access-828xl:{mountpoint:/var/lib/kubelet/pods/d961e0b7-299c-413d-8206-d7e9398e76b2/volumes/kubernetes.io~projected/kube-api-access-828xl major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db036c26-2a01-431b-a135-d3098bd05427/volumes/kubernetes.io~projected/kube-api-access-t777l:{mountpoint:/var/lib/kubelet/pods/db036c26-2a01-431b-a135-d3098bd05427/volumes/kubernetes.io~projected/kube-api-access-t777l major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db30aca3-472b-466d-be56-a6f8eb20b36a/volumes/kubernetes.io~projected/kube-api-access-mfhsn:{mountpoint:/var/lib/kubelet/pods/db30aca3-472b-466d-be56-a6f8eb20b36a/volumes/kubernetes.io~projected/kube-api-access-mfhsn major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/kube-api-access-nhh8j:{mountpoint:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/kube-api-access-nhh8j major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~projected/kube-api-access-vkncj:{mountpoint:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~projected/kube-api-access-vkncj major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f03be630-d162-45d2-8d86-6dceae239940/volumes/kubernetes.io~projected/kube-api-access-8b9kb:{mountpoint:/var/lib/kubelet/pods/f03be630-d162-45d2-8d86-6dceae239940/volumes/kubernetes.io~projected/kube-api-access-8b9kb major:0 minor:328 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9c92ad2f5fecdbb6ada0e6e55105f40e96b8b17d672b8a9b8cd43d45387dd7c1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9f54079964b52e3e24b9446700faea0a274a83640a2e8fd58031a1557e340ec3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6bf8b16a2026bd2075930e8cf7d17f57a149ad187551591e149afbfff923b5aa/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/5abf7f6ff9de857e42e9cc8962b35d9a1141bb473d36aa27cdce478773bf67b1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8e5a25f7fa80644c3a2f02d794b1c195731725c5886d11e95c8bbcb0e0b59d6b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7baec45091308320a7cb142bcf169ba639172be027deb163d4613052e31c8f05/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/46672562c17f292dc5979973789e33293d0ce9ab526c1dc58290cf4befaeec93/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e45d747c6c86bcdd2b29947c86d44fbdc05c2fd336396d931eb887594894c856/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ddcb0e94591e65a817c76ce0c6d63ea48cdd5af3013716761850f85b3f1f3a93/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/db3ff88a609f8f28bdced30b7f7c6d270a9c8e52666cbf0b122f4f7f855d31ca/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/be588c16b2180802ae6e83bf8e02a223e26e2f6c3423baa175fe139224813bb2/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/96b631e955add18350809c31879b41bc8b4b05690735fa1f6f449e3e38def0d5/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/74112ffdf50c6c331597a3dee3d679595508d1711c5e3cd8217fefc7b0e5ce39/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9d2556279585d43338a1f8970f86a47f93abe8da7748cbc71b3673a0a571ad0d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/9a4ff9cd7c74d1389689705655f36f37a07aa8aad92f544811069dfc942b051e/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/84ad6ae7a369935986a3f4cf5dcc4a11fc90a1acdeb07355f44d071b94e400a6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/02dd3e8dea3db65a5401bf33fb1bff315e472ee26d27e54f81d21c98003e6237/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e70e370f46ed11339208e40d9bba3cfc168cd4b89c2c6a9e9b51ef7990482bf0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ba6904e1bb99df0f875d57baa5beb0fff847aaba3fe54f952d8af1803a1c863e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/84b102bcac252cc89e8eb1be1985b81e75782af0f539789c099fd3f279a43b6b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5f626565b1101ce45208db81c4263f72637cce16d5fc3c711818d82eeab53a4a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b9c6b4ad7b55b18f04f730ef40a6a227e3d18aa98f410852a1b1897036c25cbf/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7e856b5c24360cd5257fef13350087d50858daf27bb283a8519da6214bd1ac95/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/72861ebf89380d57d62b95056f212ebd295f856275dde060266687e65b8f861b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/4066ef9e836e4d64140233a8070a7def592c27c4f03bf546db4754219fb3a4ae/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5b1786072bd1387dd2f1fccc2aa4dab75e16040450c6387dfbb19252a447a0d1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c46890d8edf6e6296331c9d7d6d177fa491af62832d57f1c6c66cd52b3f3ef17/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a233faabb7a8734a24778331710cf5aa9fbd3aa3efa4500a66137b7c6ffd59a8/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a9438dbcef99802776b4cc07c4eac4d9bacc9d673722fd1d5187a3298b1e10a1/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/200d373f3eab26b1a497c21cc511c2c60533df9c189125427ee6d2705d85e95f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/919b20b4f7e84592668d2cbed198acfb47613a306e775edae571e1e8c7cbc4a7/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/696ac833d8313e496a3ca74a795989110bda10865406a3c60e574c7ea2fa59fa/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/5714a4d7134dffb4c71be52e53635abdd1bbb9d4a76d13f10b98de9ad35afe9c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/e89c03ee30cf5db3321bec6d9aa522a1fb79aec66272b6d13639f5902ad26eea/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/d1701a0c64d15c3e56739b5e58db2671074674634d0a756a9370d8d34a743c66/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3392435e855047c2c5732483ccb2bb90fb8ffb298d6e35f221b136e547452803/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/6bd9338521f9cbdf6e22ee004070bce6387c8e28f15dfa13c25c3dab8e7c7868/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/48c185d471124305e2cd392f4b69634598cfd0cabc767dc81c78fccfac62b810/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/90fe6e5c93dc53102796afef5e2723bfb5c9ba3c7f0929edcd7583b1ac899702/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/33a47b2d8132e121362f0e3a1b8ae0d37e50a20278efd65995aa1de67eec1132/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/c558991b16b643afe6d902f3a4b6756fd2372e853b44c5d552041fce2e8b45f3/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/4b833a421024b1c687a90953bcb7b2011c2012d9e984fac72990b88a5efdf755/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/2096ea425468d9489793b6e4f54adade6a4122a8db687a4cd7e0308124067bbe/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/0d70d7fe32dc4c40612c14b4514dbc9ccf02a6a645bb5af10fb8af0091c9bb68/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/675bd36432bbe27d31aaa5d26ae3aa24c7a16b8976a669375ecf89b15a39a784/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5e719a069dde290b42a81c1fbc2a3c02da61ced61044c41b743a3bd18972f7fe/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/d32d8e000637107c406c4efd090bbb30f86d334cb57fe39a0ec51721a39282f0/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/cb7a5442aa3069941e8bb7f5961051c8f1765a985cb3bcbe85f1d0bb68997fbb/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/05eeae68324ac175e534c8a655a26b1fb256afa826517f673a550c13b5be4274/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/64a97bf0788b59e94a5ea1b38c4b4e62bd63059aa786830e5d2301312837d460/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/603f964a2cb506d40d01fca58f22f07c0016db415a2789089a6dc12bac869ded/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/37e213c2809995898630a9f37c11a8669b25d1924182fdb4f58240418744f5b7/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/a5c6a3e20d24d8338d59f20a58842ae328a61752f193ff602481b403f5933ef5/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/containers/storage/overlay/ac6721cf8a8ba9c871a62223618e8c19516a0a4d17d2eb67ca212beb2d866c86/merged major:0 minor:61 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f8057e778a7b852d5d756a79deefe1e6203bb08e9580bed43aa396b137464329/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/739ac68ba0492191876aef6fe814c68e3fa44178c7c042153bfc844734490ac3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/94a6b30abb25aca60a1c333a700348d0d7247fb8913dbc4169f8a5786859b2ae/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/21341db5ef5b8ba1ecee54502432fe8533620694bef9cc6c7c854025d7993f23/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/523f6e7d42efb90103d77499a52b116c9794aa5123db2d91c222239ccd8f7cb2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ec90a6473427f2c7beb3e4e763a0082f5b6affd661230d2cba5aa991fcf52bcf/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6adc272e7506ae46c6b3a5d86c4c7305c7970aa0427c4c5c086ae1bb1a916f01/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f58fd1e948e452923430f5c464a37c3a36363c8c89886af22eddc635f09b91bb/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827046    5065 manager.go:217] Machine: {Timestamp:2026-10-30 18:32:59.826596549 +0000 UTC m=+0.090061487 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150841 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b47f58ab6e323e34eae82abc2796 SystemUUID:ec24b47f-58ab-6e32-3e34-eae82abc2796 BootID:86cbab31-211f-4f83-8489-c00481e33aa3 Filesystems:[{Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~projected/kube-api-access-xn78n DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4/volumes/kubernetes.io~projected/kube-api-access-42v47 DeviceMajor:0 DeviceMinor:295 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9a74cbcd-a5c8-474b-8206-6f4b7e17b416/volumes/kubernetes.io~projected/kube-api-access-fncxb DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3392f519-fd88-4b83-bda6-f7489c047423/volumes/kubernetes.io~projected/kube-api-access-j7hkd DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/kube-api-access-hwpcn DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/200b118b-598b-4885-a4d6-f570d78aa4a7/volumes/kubernetes.io~projected/kube-api-access-zwpw6 DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/afe4c4f3-230a-4545-844b-fd72139f8e6d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~projected/kube-api-access-vkncj DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0b0ff13a73c4036ce8d6bea6e407e170f9e65bb4d8a5f2ad96cc4ca89d50c31/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/519ec4c6d4ebe0f8d1bd90bf2004362e38452db3dbb81bb6d7eb68ead0f3af3e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/658cdca61f7b857208741caf10b47a46a83821c58ba75fdb1b21eac3e8590770/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cc5c7c8156b61efdd45adf3cd3ee5ed178a2b937217117dae191d0ba0a9161f/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/db036c26-2a01-431b-a135-d3098bd05427/volumes/kubernetes.io~projected/kube-api-access-t777l DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/263e2d58-6476-493e-9de0-319a8ed0c5f7/volumes/kubernetes.io~projected/kube-api-access-rdnrp DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d83555bf-0a49-4c44-93c7-e341d040313e/volumes/kubernetes.io~projected/kube-api-access-652gv DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/156c7419-fd56-4fe4-aad9-b37b4e972a9f/volumes/kubernetes.io~projected/kube-api-access-ltfkt DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b7197c1bb05666968692759dfa6f9e0c395a29eff61f7acee2332369ccffbfa/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3fb2a473-98fc-4918-a07e-fb7d4edfddb8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/kube-api-access-nhh8j DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562/volumes/kubernetes.io~projected/kube-api-access-25bl5 DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~projected/kube-api-access-bxq7m DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~projected/kube-api-access-zksl5 DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f913778954432cdd831bc7023e78e3a749f945b2d5bc77e7f5acb3de2843ad6e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7706082d3c3dda53730a11eedc5dab1cb85e416c40733bb109bba2f40ba2c28/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c151045e-2a02-4d9e-824b-bb2d3a28902c/volumes/kubernetes.io~projected/kube-api-access-wjqd6 DeviceMajor:0 DeviceMinor:327 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8391c80c-4407-4fc4-9053-b336be68476c/volumes/kubernetes.io~projected/kube-api-access-cx5sl DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~projected/kube-api-access-c8bhv DeviceMajor:0 DeviceMinor:341 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/213646b38c9e6244dc07d60f62555b8c3aa991efa6f9e352c4d7f85e7ca35391/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/822c5b89-fcb5-4102-bf57-9ca8e15b7c40/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8c676795-0942-4bfc-8abd-65ab1071d237/volumes/kubernetes.io~projected/kube-api-access-84zfx DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b1cadbce158aa73a1dac74e59d084f1631ce38dfbaef2e59b0da1bbe9ec6b5e/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e7e558a0055118003ec5aa4c296dd55cb18fbd22f7c880f3940ea1714ad5676/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4/userdata/shm DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb49e957-8fde-4701-bbb0-b718a6acd947/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d961e0b7-299c-413d-8206-d7e9398e76b2/volumes/kubernetes.io~projected/kube-api-access-828xl DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/854afa32fcf0ab3c0ec847dd47e19174e0e39345293fa6a9e24646f3f4c46421/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b50a449-623e-44d0-bdb9-541e633dab54/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/67a5d43a-1a55-4e76-a947-629f8fe5e76e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:340 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a4411e7a-f2fb-4f3a-854c-f19b0e70b420/volumes/kubernetes.io~projected/kube-api-access-6wbvm DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/volumes/kubernetes.io~projected/kube-api-access-pt6pz DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d741136f-7a0a-479a-84ea-d74bdc485189/volumes/kubernetes.io~projected/kube-api-access-zgr5m DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/523e543cc6d33bddcaf4b72e5bcd37dcef9d0033dda59608be5648b70aa74049/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/db30aca3-472b-466d-be56-a6f8eb20b36a/volumes/kubernetes.io~projected/kube-api-access-mfhsn DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b264fe31-7f47-4667-9563-3ec98cf45e9b/volumes/kubernetes.io~projected/kube-api-access-24p4r DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24424212b883da33edd02498b09fd8b974c568a84016df0e49470c71498658b8/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fa09131-1c47-4d63-82d2-c5174c60fd49/volumes/kubernetes.io~projected/kube-api-access-l2rgf DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91140f85caf087dfb216fbe6810c53497d6811d718d3c766f294e71db32f8051/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbcc92f2cc19ce9a8da0662df454418001b9be6749d789c461e10fa80866416b/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3bc815a3-5bd4-4ec0-b417-29f36198d1be/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9ce97a9d-4f14-469c-9258-12f174cf1ac4/volumes/kubernetes.io~projected/kube-api-access-2zh6r DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0c060a2f4b4639a3ab29c312ca7e19bbf392d80ef23df00908aba2153dfeb9c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f03be630-d162-45d2-8d86-6dceae239940/volumes/kubernetes.io~projected/kube-api-access-8b9kb DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f192bcbce06187b99c864d272ee7593b20bde49b2777317608c9c09152f019b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5f9952e96c9bdc6ff2c020a3d524caa2b217df0e43345c793fbf24e17de41dd/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4/volumes/kubernetes.io~projected/kube-api-access-nqkmw DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b96f06cd6b339be66328754fdeb8e0faccffca23e75956bb7087661cf9b93d24/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5545-eaa2-487e-890b-f21a689ee08c/volumes/kubernetes.io~projected/kube-api-access-kl667 DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae7c220a2af803b26e0d78a61d65d2b2b39630aa306e7f54147ebfc3c25d337/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b5c18b284a8ac73f81666bb36c63e13125dc6a05aa648652c9b3026cf77c7af/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/061e0459b356769ed60140270cea12eab653a7efe167821bb12e939379298263/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/639b7e27be45724ab6b25c02d9a38fb4ca12a0ef56e90ad29074f6de517e4e80/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a5210ed-818c-48af-9883-0919354e4ae3/volumes/kubernetes.io~projected/kube-api-access-npxd6 DeviceMajor:0 DeviceMinor:322 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:061e0459b356769 MacAddress:f2:ac:ff:dd:57:fe Speed:10000 Mtu:8901} {Name:24424212b883da3 MacAddress:06:60:ac:23:a8:17 Speed:10000 Mtu:8901} {Name:44108dd0d2df26f MacAddress:d6:12:bd:be:43:e4 Speed:10000 Mtu:8901} {Name:4b7197c1bb05666 MacAddress:12:8a:9e:d3:36:f3 Speed:10000 Mtu:8901} {Name:4f192bcbce06187 MacAddress:e2:0c:56:d7:f2:71 Speed:10000 Mtu:8901} {Name:639b7e27be45724 MacAddress:92:3d:ef:37:67:65 Speed:10000 Mtu:8901} {Name:658cdca61f7b857 MacAddress:6e:7d:a6:f3:a0:c6 Speed:10000 Mtu:8901} {Name:7b5c18b284a8ac7 MacAddress:a6:58:16:63:eb:06 Speed:10000 Mtu:8901} {Name:7cc5c7c8156b61e MacAddress:d2:80:7b:39:ce:ca Speed:10000 Mtu:8901} {Name:8ae7c220a2af803 MacAddress:6a:d8:3b:17:b6:f1 Speed:10000 Mtu:8901} {Name:8e7e558a0055118 MacAddress:2e:c4:e1:8d:bc:bd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c2:c5:41:db Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:97:d9:37:86:dd Speed:0 Mtu:8901} {Name:c0b0ff13a73c403 MacAddress:12:4f:38:ec:fc:0a Speed:10000 Mtu:8901} {Name:d0c060a2f4b4639 MacAddress:02:c0:62:58:52:c0 Speed:10000 Mtu:8901} {Name:d5f9952e96c9bdc MacAddress:ce:a5:0a:d3:5e:cc Speed:10000 Mtu:8901} {Name:e7706082d3c3dda MacAddress:36:32:07:cb:7c:34 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c2:c5:41:db Speed:0 Mtu:9001} {Name:f913778954432cd MacAddress:c2:47:d5:04:68:92 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:9a:88:a4:6d:38 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:f0:c6:c7:4c:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827414    5065 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827454    5065 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827716    5065 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827733    5065 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-155.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827857    5065 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827864    5065 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827878    5065 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827895    5065 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.827912    5065 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828053    5065 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828164    5065 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828196    5065 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828205    5065 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828214    5065 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828222    5065 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.828230    5065 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.829019    5065 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.829029    5065 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:59.830328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.829844    5065 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:32:59.832707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.832689    5065 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:32:59.832707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.832702    5065 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:59.832844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.832807    5065 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:32:59.832994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.832981    5065 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:32:59.833049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833011    5065 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:32:59.833116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833109    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833121    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833127    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833133    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833138    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833144    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833149    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833154    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833161    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833167    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:32:59.833176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833175    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:32:59.833601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833184    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:32:59.833601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833209    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:32:59.833601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.833215    5065 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:32:59.836519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.836506    5065 server.go:1263] "Started kubelet"
Oct 30 18:32:59.836605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.836567    5065 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:32:59.836659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.836600    5065 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:32:59.836709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.836675    5065 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:32:59.837008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.836982    5065 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:32:59.837668 ip-10-0-131-155 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:32:59.837816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.837723    5065 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:32:59.839769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839748    5065 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:32:59.839769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839768    5065 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:32:59.840729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839804    5065 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:20 +0000 UTC" deadline="2028-08-10 20:58:29.0689945 +0000 UTC"
Oct 30 18:32:59.840729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839825    5065 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:32:59.840729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839835    5065 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:32:59.840729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.839820    5065 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15602h25m29.229176921s"
Oct 30 18:32:59.840729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.840323    5065 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:32:59.842639 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:32:59.839861    5065 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-155.us-east-2.compute.internal\" not found"
Oct 30 18:32:59.844739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.844048    5065 factory.go:55] Registering systemd factory
Oct 30 18:32:59.844739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.844070    5065 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:32:59.846467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846437    5065 factory.go:153] Registering CRI-O factory
Oct 30 18:32:59.846467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846457    5065 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:32:59.846624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846527    5065 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:32:59.846624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846549    5065 factory.go:103] Registering Raw factory
Oct 30 18:32:59.846624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846565    5065 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:32:59.846886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846848    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:32:59.846886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846883    5065 manager.go:319] Starting recovery of all containers
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846890    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb49e957-8fde-4701-bbb0-b718a6acd947" volumeName="kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846912    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa09131-1c47-4d63-82d2-c5174c60fd49" volumeName="kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846925    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846943    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846956    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846969    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb49e957-8fde-4701-bbb0-b718a6acd947" volumeName="kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config" seLinuxMountContext=""
Oct 30 18:32:59.846999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.846985    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200b118b-598b-4885-a4d6-f570d78aa4a7" volumeName="kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847001    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847044    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822c5b89-fcb5-4102-bf57-9ca8e15b7c40" volumeName="kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847056    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4" volumeName="kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847074    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afe4c4f3-230a-4545-844b-fd72139f8e6d" volumeName="kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847110    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d741136f-7a0a-479a-84ea-d74bdc485189" volumeName="kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847123    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db30aca3-472b-466d-be56-a6f8eb20b36a" volumeName="kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847144    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb49e957-8fde-4701-bbb0-b718a6acd947" volumeName="kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847156    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b264fe31-7f47-4667-9563-3ec98cf45e9b" volumeName="kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847172    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5545-eaa2-487e-890b-f21a689ee08c" volumeName="kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847184    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5545-eaa2-487e-890b-f21a689ee08c" volumeName="kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847200    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="263e2d58-6476-493e-9de0-319a8ed0c5f7" volumeName="kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847211    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3392f519-fd88-4b83-bda6-f7489c047423" volumeName="kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847223    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fb2a473-98fc-4918-a07e-fb7d4edfddb8" volumeName="kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847240    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847253    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8391c80c-4407-4fc4-9053-b336be68476c" volumeName="kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847270    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847282    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a5d43a-1a55-4e76-a947-629f8fe5e76e" volumeName="kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847294    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5210ed-818c-48af-9883-0919354e4ae3" volumeName="kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847310    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f03be630-d162-45d2-8d86-6dceae239940" volumeName="kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847322    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f03be630-d162-45d2-8d86-6dceae239940" volumeName="kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847348    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="263e2d58-6476-493e-9de0-319a8ed0c5f7" volumeName="kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847365    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847378    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847390    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" volumeName="kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847406    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" volumeName="kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb" seLinuxMountContext=""
Oct 30 18:32:59.847406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847419    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb49e957-8fde-4701-bbb0-b718a6acd947" volumeName="kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847435    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb49e957-8fde-4701-bbb0-b718a6acd947" volumeName="kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847446    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847459    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5210ed-818c-48af-9883-0919354e4ae3" volumeName="kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847475    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c676795-0942-4bfc-8abd-65ab1071d237" volumeName="kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847490    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847505    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5545-eaa2-487e-890b-f21a689ee08c" volumeName="kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847517    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847530    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4411e7a-f2fb-4f3a-854c-f19b0e70b420" volumeName="kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847549    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d938f8-d5bf-42a9-b4c5-169b0f0d6777" volumeName="kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847561    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847579    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5210ed-818c-48af-9883-0919354e4ae3" volumeName="kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847593    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3392f519-fd88-4b83-bda6-f7489c047423" volumeName="kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847604    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc815a3-5bd4-4ec0-b417-29f36198d1be" volumeName="kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847628    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5210ed-818c-48af-9883-0919354e4ae3" volumeName="kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847641    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8391c80c-4407-4fc4-9053-b336be68476c" volumeName="kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847660    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="156c7419-fd56-4fe4-aad9-b37b4e972a9f" volumeName="kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847673    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c676795-0942-4bfc-8abd-65ab1071d237" volumeName="kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847685    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c676795-0942-4bfc-8abd-65ab1071d237" volumeName="kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847700    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db30aca3-472b-466d-be56-a6f8eb20b36a" volumeName="kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847712    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847731    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847743    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa09131-1c47-4d63-82d2-c5174c60fd49" volumeName="kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847773    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847789    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fb2a473-98fc-4918-a07e-fb7d4edfddb8" volumeName="kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847801    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822c5b89-fcb5-4102-bf57-9ca8e15b7c40" volumeName="kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847819    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8391c80c-4407-4fc4-9053-b336be68476c" volumeName="kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847833    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c151045e-2a02-4d9e-824b-bb2d3a28902c" volumeName="kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847844    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f03be630-d162-45d2-8d86-6dceae239940" volumeName="kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847860    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3392f519-fd88-4b83-bda6-f7489c047423" volumeName="kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847873    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a5d43a-1a55-4e76-a947-629f8fe5e76e" volumeName="kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847890    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce97a9d-4f14-469c-9258-12f174cf1ac4" volumeName="kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847907    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4" volumeName="kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847920    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b50a449-623e-44d0-bdb9-541e633dab54" volumeName="kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847939    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc815a3-5bd4-4ec0-b417-29f36198d1be" volumeName="kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847952    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562" volumeName="kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847969    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847982    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" volumeName="kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.847996    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4411e7a-f2fb-4f3a-854c-f19b0e70b420" volumeName="kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848011    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d961e0b7-299c-413d-8206-d7e9398e76b2" volumeName="kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848024    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2" volumeName="kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848040    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2" volumeName="kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848053    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bc815a3-5bd4-4ec0-b417-29f36198d1be" volumeName="kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848065    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a5d43a-1a55-4e76-a947-629f8fe5e76e" volumeName="kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848100    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" volumeName="kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848114    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4411e7a-f2fb-4f3a-854c-f19b0e70b420" volumeName="kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848130    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b264fe31-7f47-4667-9563-3ec98cf45e9b" volumeName="kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848142    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d741136f-7a0a-479a-84ea-d74bdc485189" volumeName="kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848154    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5545-eaa2-487e-890b-f21a689ee08c" volumeName="kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848169    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c39f772-3455-4174-a1b8-49b8a5f74b30" volumeName="kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848180    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="822c5b89-fcb5-4102-bf57-9ca8e15b7c40" volumeName="kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848196    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83555bf-0a49-4c44-93c7-e341d040313e" volumeName="kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848208    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db036c26-2a01-431b-a135-d3098bd05427" volumeName="kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848220    5065 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de844348-2c71-4773-b997-e1a79da8e4e1" volumeName="kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token" seLinuxMountContext=""
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848235    5065 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:32:59.848716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.848242    5065 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:32:59.858912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.858470    5065 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:59.858912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.858699    5065 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:59.860898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.860864    5065 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:32:59.860991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.860929    5065 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:59.868027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.868009    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:32:59.868616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.868460    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:32:59.877756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.877736    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:59.877844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.877764    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:59.879604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.879572    5065 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:32:59.881167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.881149    5065 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:32:59.881261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.881186    5065 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:32:59.881261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.881207    5065 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:32:59.881506 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:32:59.881456    5065 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:32:59.885859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.885838    5065 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:59.901070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901048    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="c15b14ad18eb9129b5893c5ba67686c26fac0cf0c66985f0eb594120c5f4f4fc" exitCode=0
Oct 30 18:32:59.901070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901068    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="d1fd27117c47bb926ad8fb7c97500e79177bb1a9f4d0e59762fb38f7f25d3ef2" exitCode=0
Oct 30 18:32:59.901231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901074    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="a8eac129431479bcb1d722bd8194a8caccfdc42b1468ccb4ab419824ca9617d4" exitCode=0
Oct 30 18:32:59.901231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901080    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="d293e61dfb770ac07c3cacbff87235fbc4cb6b26cbd42b411a9f901f5b94d2c8" exitCode=0
Oct 30 18:32:59.901231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901104    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="00c863b0936cf8c4ee6881ab746fcab346e9f231bef57b548619afb704ee5386" exitCode=0
Oct 30 18:32:59.901231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.901109    5065 generic.go:356] "Generic (PLEG): container finished" podID="9a74cbcd-a5c8-474b-8206-6f4b7e17b416" containerID="14f81edd5c5895968e62cd679a45c01ab2955d8016d900a8e58fbeeb820c5130" exitCode=0
Oct 30 18:32:59.919707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.919688    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:32:59.920044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.920020    5065 generic.go:356] "Generic (PLEG): container finished" podID="eb49e957-8fde-4701-bbb0-b718a6acd947" containerID="5b90f94b53a3c60b4a35d0b02adbd8310842582d2bcc75b9db4c1de0fc5571d8" exitCode=1
Oct 30 18:32:59.922184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.922131    5065 generic.go:356] "Generic (PLEG): container finished" podID="210591172398def826e79b175b0190f4" containerID="ac284a4bb6fd3802542259a3ec8453625147486172246eb043cade75e6772593" exitCode=0
Oct 30 18:32:59.925482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.925463    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:32:59.925594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.925493    5065 generic.go:356] "Generic (PLEG): container finished" podID="0fab5545-eaa2-487e-890b-f21a689ee08c" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492" exitCode=255
Oct 30 18:32:59.946190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.946174    5065 manager.go:324] Recovery completed
Oct 30 18:32:59.966786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966765    5065 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:32:59.966786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966780    5065 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:32:59.966901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966800    5065 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:32:59.966939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966923    5065 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:32:59.966970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966940    5065 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:32:59.966970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966956    5065 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:32:59.966970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966961    5065 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:32:59.966970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966966    5065 policy_none.go:50] "Start"
Oct 30 18:32:59.967078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966974    5065 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:32:59.967078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.966983    5065 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:32:59.967143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.967115    5065 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:32:59.967143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.967122    5065 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:32:59.967143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.967127    5065 policy_none.go:44] "Start"
Oct 30 18:32:59.972863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.972827    5065 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:32:59.975415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.975400    5065 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:32:59.975960 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:32:59.975429    5065 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:32:59.976016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.975964    5065 server.go:87] "Starting device plugin registration server"
Oct 30 18:32:59.976240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976227    5065 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:32:59.976273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976244    5065 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:32:59.976368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976346    5065 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:32:59.976485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976438    5065 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:32:59.976485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976477    5065 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:32:59.976485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976484    5065 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:32:59.976681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.976515    5065 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:32:59.976466164Z","UUID":"ebbee2e4-0755-4bfb-8fcc-dc849bd8f6be","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:32:59.977842 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:32:59.977823    5065 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:32:59.977967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.977947    5065 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:32:59.978041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.977977    5065 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:32:59.981575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.981535    5065 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"]
Oct 30 18:32:59.982282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982267    5065 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:32:59.982382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982341    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerStarted","Data":"92f6880441819cf1baddaa17e1726704e47d5b23e5899dc8bd4933b50eafd277"}
Oct 30 18:32:59.982382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982378    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerDied","Data":"ac284a4bb6fd3802542259a3ec8453625147486172246eb043cade75e6772593"}
Oct 30 18:32:59.982505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982395    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"210591172398def826e79b175b0190f4","Type":"ContainerStarted","Data":"c748aef02c43505eff180cc35beb2b98249aa452de13f042c9eaf37f43839fa4"}
Oct 30 18:32:59.982505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982441    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"dabecda5632cef959a155a21c4c95263","Type":"ContainerStarted","Data":"90a3f282cb587398a3f40f052eda6fb966888f7e822cb9973fa326d5281ce770"}
Oct 30 18:32:59.982505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982462    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal" event={"ID":"dabecda5632cef959a155a21c4c95263","Type":"ContainerStarted","Data":"cc5733c572d35a3db4d90b2e9b8d61d0466611c7d2585c8adfe62b2a0db993b2"}
Oct 30 18:32:59.982664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:32:59.982589    5065 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.010602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.010581    5065 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:00.010692 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:00.010640    5065 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.010769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.010746    5065 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:00.010870 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:00.010798    5065 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.062156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.062122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.062260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.062160    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.062260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.062182    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.077276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.077255    5065 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:33:00.078360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.078346    5065 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:33:00.078428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.078374    5065 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:33:00.078428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.078389    5065 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-155.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:33:00.078483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.078439    5065 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.097233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.097216    5065 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.097280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.097268    5065 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163217    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163251    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163278    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163317    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163326    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabecda5632cef959a155a21c4c95263-config\") pod \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"dabecda5632cef959a155a21c4c95263\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.163526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.163326    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210591172398def826e79b175b0190f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" (UID: \"210591172398def826e79b175b0190f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.828658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.828635    5065 apiserver.go:52] "Watching apiserver"
Oct 30 18:33:00.842098 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.842060    5065 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:33:00.842886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.842861    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm","openshift-image-registry/image-registry-5b594796db-ldb2p","kube-system/kas-connection-checker-685895c7c9-whf9r","kube-system/global-pull-secret-syncer-lg2zm","kube-system/kas-connection-checker-685895c7c9-hghfk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8","openshift-image-registry/node-ca-fqpv4","openshift-multus/network-metrics-daemon-c5n4z","openshift-ingress-canary/ingress-canary-wrlwc","openshift-insights/insights-runtime-extractor-hvfzd","openshift-multus/multus-g2z97","openshift-dns/dns-default-l8bh7","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76","openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6","kube-system/kas-connection-checker-685895c7c9-vczpg","openshift-multus/multus-additional-cni-plugins-q9pks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc","openshift-network-diagnostics/network-check-target-w7prc","openshift-insights/insights-operator-7c4986d449-ghhlw","openshift-console-operator/console-operator-7cd56b8dbc-rsttd","openshift-cluster-node-tuning-operator/tuned-wpghw","openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj","openshift-image-registry/image-registry-59c4bf777f-scwmb","openshift-network-operator/iptables-alerter-tnzkp","openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct","kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-9lr9k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9","openshift-ingress/router-default-56f55855fc-ctkk9","openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w","openshift-service-ca/service-ca-5fffc97bdf-jqr4n","openshift-network-console/networking-console-plugin-86c478bb88-gjzjs","kube-system/konnectivity-agent-nhsc4","openshift-dns/node-resolver-prl4g","openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"]
Oct 30 18:33:00.844103 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.844071    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:00.844408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.844386    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.846838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.846821    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.847322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.847304    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.847775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.847758    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:00.854039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.854020    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:00.856293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.856274    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:33:00.856752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.856738    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.857513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.857492    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.857720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.857704    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.858472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.858454    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.859692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.859669    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.860502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860484    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.860615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860597    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.860706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860689    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.860760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860691    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:33:00.860760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860719    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.860844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860777    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.860844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860835    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860934    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860943    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860954    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860936    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860954    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.860977    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:33:00.861340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861014    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.861340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861018    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:33:00.861637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861624    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861635    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:33:00.861732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861693    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.861732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861728    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861735    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.861816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861745    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.861816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.861750    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.862772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.862760    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.862967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.862951    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:33:00.863006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.862957    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:33:00.863199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.863183    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.868396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.868105    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:33:00.873250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.873227    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:33:00.873346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.873318    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:33:00.875442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.875422    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.875751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.875730    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:33:00.876375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.876355    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.876589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.876573    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:33:00.876667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.876641    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:33:00.876896 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.876880    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:33:00.877107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877075    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:33:00.877223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877204    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:33:00.877297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877280    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:33:00.877392 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877360    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:33:00.877501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877418    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:33:00.877553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877536    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:33:00.877783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877758    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:33:00.877925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877895    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lvtf9\""
Oct 30 18:33:00.877925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.877909    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j8xl4\""
Oct 30 18:33:00.878167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878144    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:33:00.878254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878236    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\""
Oct 30 18:33:00.878308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878276    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.878369 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878313    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:33:00.878414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878405    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.878501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878487    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:33:00.878615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878595    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:33:00.878670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878630    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2fw8l\""
Oct 30 18:33:00.878738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878725    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2hpwn\""
Oct 30 18:33:00.878847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.878833    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:33:00.879409 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879390    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-5wnwl\""
Oct 30 18:33:00.879506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879459    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\""
Oct 30 18:33:00.879620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879399    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\""
Oct 30 18:33:00.879761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879685    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n52cc\""
Oct 30 18:33:00.879761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879757    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:33:00.879861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879804    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zs4rg\""
Oct 30 18:33:00.879944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879929    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7b9zd\""
Oct 30 18:33:00.879998 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.879950    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\""
Oct 30 18:33:00.880054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880031    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:33:00.880054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880040    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\""
Oct 30 18:33:00.880180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880061    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:33:00.880744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880728    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g4tgh\""
Oct 30 18:33:00.880837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880756    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\""
Oct 30 18:33:00.880837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880821    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.880837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880828    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-glmtq\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880844    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880849    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880860    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g8dfk\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880868    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880887    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880891    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880915    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880934    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880938    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880950    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880972    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880977    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.880997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880991    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880990    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880992    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881025    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880936    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881026    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.880972    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881044    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881048    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881102    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881128    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881148    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881156    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881183    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881207    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:00.881505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.881214    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\""
Oct 30 18:33:00.942126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.942110    5065 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:33:00.945363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.945347    5065 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.945503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.945490    5065 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.959726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.959707    5065 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:00.959793 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:00.959745    5065 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.960297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.960276    5065 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:00.960357 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:00.960312    5065 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-155.us-east-2.compute.internal"
Oct 30 18:33:00.964848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.964831    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:33:00.965133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965114    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:33:00.965286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965170    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\""
Oct 30 18:33:00.965286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965204    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965314    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965332    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6hqgv\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965333    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6twlp\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965347    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqrtm\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965353    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:33:00.965398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965376    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bmnq2\""
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965400    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\""
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965409    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\""
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965453    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pg8bd\""
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965492    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bl5\" (UniqueName: \"kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5\") pod \"kas-connection-checker-685895c7c9-whf9r\" (UID: \"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562\") " pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965506    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\""
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965513    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkmw\" (UniqueName: \"kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw\") pod \"kas-connection-checker-685895c7c9-vczpg\" (UID: \"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4\") " pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965529    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr5m\" (UniqueName: \"kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965547    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.965657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965564    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpw6\" (UniqueName: \"kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6\") pod \"kas-connection-checker-685895c7c9-hghfk\" (UID: \"200b118b-598b-4885-a4d6-f570d78aa4a7\") " pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965705    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhsn\" (UniqueName: \"kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965732    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqd6\" (UniqueName: \"kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6\") pod \"migrator-66df5d496b-jdzz6\" (UID: \"c151045e-2a02-4d9e-824b-bb2d3a28902c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965761    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965778    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965794    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b264fe31-7f47-4667-9563-3ec98cf45e9b-cert\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965790    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965881    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5sl\" (UniqueName: \"kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965902    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652gv\" (UniqueName: \"kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv\") pod \"migrator-66df5d496b-2rn9w\" (UID: \"d83555bf-0a49-4c44-93c7-e341d040313e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965945    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965967    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbvm\" (UniqueName: \"kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.965998    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966024    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:00.966051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966053    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966054    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-tuned\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966129    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966156    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966168    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c676795-0942-4bfc-8abd-65ab1071d237-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966173    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966208    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hkd\" (UniqueName: \"kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966289    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966306    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966328    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966385    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966396    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966442    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966472    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966510    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxq7m\" (UniqueName: \"kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966533    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-script-lib\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966550    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966536    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966584    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966616    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966631    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966669    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966683    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c676795-0942-4bfc-8abd-65ab1071d237-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966705    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.966738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966739    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966772    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966822    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966871    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966918    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966985    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.966992    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967035    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967060    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bhv\" (UniqueName: \"kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967060    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-stats-auth\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967077    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967122    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfkt\" (UniqueName: \"kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt\") pod \"network-check-source-6f78d59f99-t9ncn\" (UID: \"156c7419-fd56-4fe4-aad9-b37b4e972a9f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967175    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb49e957-8fde-4701-bbb0-b718a6acd947-ovn-node-metrics-cert\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967213    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967239    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967273    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967353    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967374    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967390    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967406    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967424    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967466    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-multus-daemon-config\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967543    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967564    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967601    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t777l\" (UniqueName: \"kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967646    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967706    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967730    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967779    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.967803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967786    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967836    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967972    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967999    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968015    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968033    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.967998    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968043    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968109    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968120    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968157    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968188    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968216    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968229    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968244    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24p4r\" (UniqueName: \"kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968190    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-tmp\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968266    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-serving-cert\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968267    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968300    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968338    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968355    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968372    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968379    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8391c80c-4407-4fc4-9053-b336be68476c-config-volume\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968413    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968428    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968444    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968480    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968497    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6pz\" (UniqueName: \"kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968511    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968564    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968589    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zfx\" (UniqueName: \"kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968595    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-default-certificate\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968726    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968724    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-env-overrides\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968784    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03be630-d162-45d2-8d86-6dceae239940-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968807    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9kb\" (UniqueName: \"kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968831    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968884    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968949    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-trusted-ca\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.968989    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969018    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksl5\" (UniqueName: \"kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969078    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fab5545-eaa2-487e-890b-f21a689ee08c-serving-cert\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969121    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.969162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969160    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969326    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969342    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969376    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969407    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969441    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969498    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828xl\" (UniqueName: \"kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969526    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969555    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969577    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-config\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969608    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969632    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969648    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969661    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969698    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969723    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969728    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969749    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969764    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969830    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8391c80c-4407-4fc4-9053-b336be68476c-tmp-dir\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969830    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969870    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.969947    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970041    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970075    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a5d43a-1a55-4e76-a947-629f8fe5e76e-signing-key\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970114    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d741136f-7a0a-479a-84ea-d74bdc485189-iptables-alerter-script\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970128    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970147    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970162    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970208    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnrp\" (UniqueName: \"kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970283    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/263e2d58-6476-493e-9de0-319a8ed0c5f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970294    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970358    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970386    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe4c4f3-230a-4545-844b-fd72139f8e6d-original-pull-secret\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970388    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970422    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970454    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970482    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970488    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rgf\" (UniqueName: \"kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970506    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn78n\" (UniqueName: \"kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970514    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-tmp-dir\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b50a449-623e-44d0-bdb9-541e633dab54-serving-cert\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970523    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970570    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970599    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970627    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970657    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970685    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970712    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970732    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03be630-d162-45d2-8d86-6dceae239940-data-volume\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970744    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:00.970792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970812    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb49e957-8fde-4701-bbb0-b718a6acd947-ovnkube-config\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970864    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3392f519-fd88-4b83-bda6-f7489c047423-cni-binary-copy\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970880    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970906    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970934    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.970977    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971010    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971018    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971025    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971042    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh6r\" (UniqueName: \"kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r\") pod \"volume-data-source-validator-58c9f5f46d-ntj76\" (UID: \"9ce97a9d-4f14-469c-9258-12f174cf1ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971071    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971136    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971173    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971226    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971266    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-agent-certs\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971269    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971298    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncxb\" (UniqueName: \"kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971325    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971332    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fb2a473-98fc-4918-a07e-fb7d4edfddb8-konnectivity-ca\") pod \"konnectivity-agent-nhsc4\" (UID: \"3fb2a473-98fc-4918-a07e-fb7d4edfddb8\") " pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971342    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl667\" (UniqueName: \"kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971379    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971394    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b50a449-623e-44d0-bdb9-541e633dab54-service-ca-bundle\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971490    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971522    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971571    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b50a449-623e-44d0-bdb9-541e633dab54-snapshots\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971573    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971597    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971621    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971637    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971666    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971714    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-tmp\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.971863    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0fab5545-eaa2-487e-890b-f21a689ee08c-config\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:00.972575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.972077    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fa09131-1c47-4d63-82d2-c5174c60fd49-serviceca\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:00.978742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.978724    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhsn\" (UniqueName: \"kubernetes.io/projected/db30aca3-472b-466d-be56-a6f8eb20b36a-kube-api-access-mfhsn\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:00.982015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.981999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr5m\" (UniqueName: \"kubernetes.io/projected/d741136f-7a0a-479a-84ea-d74bdc485189-kube-api-access-zgr5m\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:00.990596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.990581    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5sl\" (UniqueName: \"kubernetes.io/projected/8391c80c-4407-4fc4-9053-b336be68476c-kube-api-access-cx5sl\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:00.991606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.991591    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkmw\" (UniqueName: \"kubernetes.io/projected/a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4-kube-api-access-nqkmw\") pod \"kas-connection-checker-685895c7c9-vczpg\" (UID: \"a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4\") " pod="kube-system/kas-connection-checker-685895c7c9-vczpg"
Oct 30 18:33:00.991678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.991664    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hkd\" (UniqueName: \"kubernetes.io/projected/3392f519-fd88-4b83-bda6-f7489c047423-kube-api-access-j7hkd\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:00.995432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.995413    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bl5\" (UniqueName: \"kubernetes.io/projected/3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562-kube-api-access-25bl5\") pod \"kas-connection-checker-685895c7c9-whf9r\" (UID: \"3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562\") " pod="kube-system/kas-connection-checker-685895c7c9-whf9r"
Oct 30 18:33:00.995538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:00.995525    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh6r\" (UniqueName: \"kubernetes.io/projected/9ce97a9d-4f14-469c-9258-12f174cf1ac4-kube-api-access-2zh6r\") pod \"volume-data-source-validator-58c9f5f46d-ntj76\" (UID: \"9ce97a9d-4f14-469c-9258-12f174cf1ac4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ntj76"
Oct 30 18:33:01.008930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.008903    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpw6\" (UniqueName: \"kubernetes.io/projected/200b118b-598b-4885-a4d6-f570d78aa4a7-kube-api-access-zwpw6\") pod \"kas-connection-checker-685895c7c9-hghfk\" (UID: \"200b118b-598b-4885-a4d6-f570d78aa4a7\") " pod="kube-system/kas-connection-checker-685895c7c9-hghfk"
Oct 30 18:33:01.009844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.009823    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkncj\" (UniqueName: \"kubernetes.io/projected/eb49e957-8fde-4701-bbb0-b718a6acd947-kube-api-access-vkncj\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.009844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.009833    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bhv\" (UniqueName: \"kubernetes.io/projected/67a5d43a-1a55-4e76-a947-629f8fe5e76e-kube-api-access-c8bhv\") pod \"service-ca-5fffc97bdf-jqr4n\" (UID: \"67a5d43a-1a55-4e76-a947-629f8fe5e76e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jqr4n"
Oct 30 18:33:01.009999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.009962    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652gv\" (UniqueName: \"kubernetes.io/projected/d83555bf-0a49-4c44-93c7-e341d040313e-kube-api-access-652gv\") pod \"migrator-66df5d496b-2rn9w\" (UID: \"d83555bf-0a49-4c44-93c7-e341d040313e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rn9w"
Oct 30 18:33:01.010299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010280    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t777l\" (UniqueName: \"kubernetes.io/projected/db036c26-2a01-431b-a135-d3098bd05427-kube-api-access-t777l\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:01.010389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010285    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbvm\" (UniqueName: \"kubernetes.io/projected/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-kube-api-access-6wbvm\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.010480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010461    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v47\" (UniqueName: \"kubernetes.io/projected/1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4-kube-api-access-42v47\") pod \"network-check-target-w7prc\" (UID: \"1fe8a4de-10ce-4d60-8f5d-bce0376fa5d4\") " pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:33:01.010567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010478    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxq7m\" (UniqueName: \"kubernetes.io/projected/822c5b89-fcb5-4102-bf57-9ca8e15b7c40-kube-api-access-bxq7m\") pod \"service-ca-operator-544d749c8c-6klqj\" (UID: \"822c5b89-fcb5-4102-bf57-9ca8e15b7c40\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj"
Oct 30 18:33:01.010567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010478    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksl5\" (UniqueName: \"kubernetes.io/projected/3b50a449-623e-44d0-bdb9-541e633dab54-kube-api-access-zksl5\") pod \"insights-operator-7c4986d449-ghhlw\" (UID: \"3b50a449-623e-44d0-bdb9-541e633dab54\") " pod="openshift-insights/insights-operator-7c4986d449-ghhlw"
Oct 30 18:33:01.010567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010533    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:01.010567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010541    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfkt\" (UniqueName: \"kubernetes.io/projected/156c7419-fd56-4fe4-aad9-b37b4e972a9f-kube-api-access-ltfkt\") pod \"network-check-source-6f78d59f99-t9ncn\" (UID: \"156c7419-fd56-4fe4-aad9-b37b4e972a9f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-t9ncn"
Oct 30 18:33:01.010861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010609    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zfx\" (UniqueName: \"kubernetes.io/projected/8c676795-0942-4bfc-8abd-65ab1071d237-kube-api-access-84zfx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-llkhc\" (UID: \"8c676795-0942-4bfc-8abd-65ab1071d237\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc"
Oct 30 18:33:01.010861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010747    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqd6\" (UniqueName: \"kubernetes.io/projected/c151045e-2a02-4d9e-824b-bb2d3a28902c-kube-api-access-wjqd6\") pod \"migrator-66df5d496b-jdzz6\" (UID: \"c151045e-2a02-4d9e-824b-bb2d3a28902c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jdzz6"
Oct 30 18:33:01.011015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010959    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:01.011015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.010989    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9kb\" (UniqueName: \"kubernetes.io/projected/f03be630-d162-45d2-8d86-6dceae239940-kube-api-access-8b9kb\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:01.011284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011259    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl667\" (UniqueName: \"kubernetes.io/projected/0fab5545-eaa2-487e-890b-f21a689ee08c-kube-api-access-kl667\") pod \"console-operator-7cd56b8dbc-rsttd\" (UID: \"0fab5545-eaa2-487e-890b-f21a689ee08c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:01.011367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011281    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:01.011367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011295    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"periodic-gathering-glkck-6hcwm\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") " pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:01.011535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011513    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnrp\" (UniqueName: \"kubernetes.io/projected/263e2d58-6476-493e-9de0-319a8ed0c5f7-kube-api-access-rdnrp\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:01.011626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011565    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828xl\" (UniqueName: \"kubernetes.io/projected/d961e0b7-299c-413d-8206-d7e9398e76b2-kube-api-access-828xl\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.011755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011734    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6pz\" (UniqueName: \"kubernetes.io/projected/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-kube-api-access-pt6pz\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:01.011832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011772    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:01.011832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.011790    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncxb\" (UniqueName: \"kubernetes.io/projected/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-kube-api-access-fncxb\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.012066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.012045    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rgf\" (UniqueName: \"kubernetes.io/projected/2fa09131-1c47-4d63-82d2-c5174c60fd49-kube-api-access-l2rgf\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:01.012135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.012055    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn78n\" (UniqueName: \"kubernetes.io/projected/3bc815a3-5bd4-4ec0-b417-29f36198d1be-kube-api-access-xn78n\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:01.015553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.015537    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24p4r\" (UniqueName: \"kubernetes.io/projected/b264fe31-7f47-4667-9563-3ec98cf45e9b-kube-api-access-24p4r\") pod \"ingress-canary-wrlwc\" (UID: \"b264fe31-7f47-4667-9563-3ec98cf45e9b\") " pod="openshift-ingress-canary/ingress-canary-wrlwc"
Oct 30 18:33:01.072021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.071995    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.072021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072023    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072040    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072056    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072070    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072136    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-bin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-cnibin\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072148    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072161    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:01.072202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072194    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072232    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-etc-kubernetes\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072240    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072268    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072276    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-socket-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072292    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072311    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2-hosts-file\") pod \"node-resolver-prl4g\" (UID: \"0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2\") " pod="openshift-dns/node-resolver-prl4g"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072317    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072337    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072340    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072360    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072385    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072392    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysconfig\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072408    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072437    5065 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072459    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-kubernetes\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072465    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-run\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072506    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072538    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072560    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.072577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072587    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072568    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-device-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072614    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072641    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072645    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-etc-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072663    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072692    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072703    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-systemd\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072714    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072741    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-bin\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072791    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.072839    5065 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072847    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03be630-d162-45d2-8d86-6dceae239940-crio-socket\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-kubelet-config\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.072930    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:01.572911366 +0000 UTC m=+1.836376307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072937    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-netns\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072980    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-system-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.072740    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-hostroot\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073026    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073051    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073125    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-conf-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073263    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073334    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-run-netns\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073291    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.073370    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:33:01.573353533 +0000 UTC m=+1.836818477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.073475    5065 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073510    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.073537    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:33:01.573522351 +0000 UTC m=+1.836987293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073565    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073588    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073615    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073638    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073658    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-modprobe-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073668    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073699    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073720    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073743    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.073746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073765    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073790    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073801    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-conf\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073813    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073872    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073879    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-socket-dir-parent\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073908    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-multus-certs\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073912    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073957    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.073981    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074019    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074044    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074066    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074126    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074148    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074182    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-slash\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074224    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d741136f-7a0a-479a-84ea-d74bdc485189-host-slash\") pod \"iptables-alerter-tnzkp\" (UID: \"d741136f-7a0a-479a-84ea-d74bdc485189\") " pod="openshift-network-operator/iptables-alerter-tnzkp"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074229    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074268    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-cnibin\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-host\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074302    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-systemd-units\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074338    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fa09131-1c47-4d63-82d2-c5174c60fd49-host\") pod \"node-ca-fqpv4\" (UID: \"2fa09131-1c47-4d63-82d2-c5174c60fd49\") " pod="openshift-image-registry/node-ca-fqpv4"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074169    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-sys\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074353    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-kubelet\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074370    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074395    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074403    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-sys-fs\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074419    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074450    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074459    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074484    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-kubelet\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074485    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074526    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074558    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074617    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074660    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074690    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074712    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074750    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074774    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074796    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074846    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe4c4f3-230a-4545-844b-fd72139f8e6d-dbus\") pod \"global-pull-secret-syncer-lg2zm\" (UID: \"afe4c4f3-230a-4545-844b-fd72139f8e6d\") " pod="kube-system/global-pull-secret-syncer-lg2zm"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074900    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-multus-cni-dir\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074930    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d961e0b7-299c-413d-8206-d7e9398e76b2-registration-dir\") pod \"aws-ebs-csi-driver-node-pxqg9\" (UID: \"d961e0b7-299c-413d-8206-d7e9398e76b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxqg9"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074935    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-log-socket\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.074989    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-host-cni-netd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.075018    5065 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075055    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.075067    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:01.57505429 +0000 UTC m=+1.838519230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075119    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-systemd\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075123    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-run-ovn\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075160    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-node-log\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075163    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a74cbcd-a5c8-474b-8206-6f4b7e17b416-system-cni-dir\") pod \"multus-additional-cni-plugins-q9pks\" (UID: \"9a74cbcd-a5c8-474b-8206-6f4b7e17b416\") " pod="openshift-multus/multus-additional-cni-plugins-q9pks"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075210    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-var-lib-cni-multus\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075210    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-var-lib-kubelet\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075257    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb49e957-8fde-4701-bbb0-b718a6acd947-var-lib-openvswitch\") pod \"ovnkube-node-9lr9k\" (UID: \"eb49e957-8fde-4701-bbb0-b718a6acd947\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3392f519-fd88-4b83-bda6-f7489c047423-host-run-k8s-cni-cncf-io\") pod \"multus-g2z97\" (UID: \"3392f519-fd88-4b83-bda6-f7489c047423\") " pod="openshift-multus/multus-g2z97"
Oct 30 18:33:01.075307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075308    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-lib-modules\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075392    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4411e7a-f2fb-4f3a-854c-f19b0e70b420-etc-sysctl-d\") pod \"tuned-wpghw\" (UID: \"a4411e7a-f2fb-4f3a-854c-f19b0e70b420\") " pod="openshift-cluster-node-tuning-operator/tuned-wpghw"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075372    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"image-registry-5b594796db-ldb2p\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") " pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075473    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/263e2d58-6476-493e-9de0-319a8ed0c5f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lf2ct\" (UID: \"263e2d58-6476-493e-9de0-319a8ed0c5f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075623    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db036c26-2a01-431b-a135-d3098bd05427-metrics-certs\") pod \"network-metrics-daemon-c5n4z\" (UID: \"db036c26-2a01-431b-a135-d3098bd05427\") " pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.075999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bc815a3-5bd4-4ec0-b417-29f36198d1be-metrics-certs\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:01.077898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.077676    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8391c80c-4407-4fc4-9053-b336be68476c-metrics-tls\") pod \"dns-default-l8bh7\" (UID: \"8391c80c-4407-4fc4-9053-b336be68476c\") " pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:01.078108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.077987    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"image-registry-59c4bf777f-scwmb\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") " pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:01.144385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.144341    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"
Oct 30 18:33:01.146684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.146664    5065 scope.go:122] "RemoveContainer" containerID="5f29b98829c1ed58b41a8bba31e32f92739396177dadda59392b2f246f03f492"
Oct 30 18:33:01.147069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.147055    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:01.154527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.154500    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5n4z"
Oct 30 18:33:01.157666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.157649    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:01.158804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.158744    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:01.474335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.474302    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct"]
Oct 30 18:33:01.478548 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:01.478521    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263e2d58_6476_493e_9de0_319a8ed0c5f7.slice/crio-9632f0fda9e3c87831236876cdc13d691b6cba58144c35592424d9165c5ca284 WatchSource:0}: Error finding container 9632f0fda9e3c87831236876cdc13d691b6cba58144c35592424d9165c5ca284: Status 404 returned error can't find the container with id 9632f0fda9e3c87831236876cdc13d691b6cba58144c35592424d9165c5ca284
Oct 30 18:33:01.481215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.481200    5065 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:33:01.488878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.488855    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:01.521406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.521367    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:33:01.529258 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:01.529228    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c39f772_3455_4174_a1b8_49b8a5f74b30.slice/crio-1cd8e2a65f96297698aaa51a63122100887f65384a316237371e8206e91c64a4 WatchSource:0}: Error finding container 1cd8e2a65f96297698aaa51a63122100887f65384a316237371e8206e91c64a4: Status 404 returned error can't find the container with id 1cd8e2a65f96297698aaa51a63122100887f65384a316237371e8206e91c64a4
Oct 30 18:33:01.574059 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.573926    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8bh7"]
Oct 30 18:33:01.577172 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:01.576051    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8391c80c_4407_4fc4_9053_b336be68476c.slice/crio-da9b359683d86e2ae317117f23ae0329f98ffaeba66a63d0a795b54fbb6f2572 WatchSource:0}: Error finding container da9b359683d86e2ae317117f23ae0329f98ffaeba66a63d0a795b54fbb6f2572: Status 404 returned error can't find the container with id da9b359683d86e2ae317117f23ae0329f98ffaeba66a63d0a795b54fbb6f2572
Oct 30 18:33:01.577172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.576962    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5n4z"]
Oct 30 18:33:01.579344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.579324    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:01.579658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.579394    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:01.579658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.579439    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:01.579658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.579474    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.579745    5065 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.579799    5065 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.579897    5065 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.579823    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle podName:3bc815a3-5bd4-4ec0-b417-29f36198d1be nodeName:}" failed. No retries permitted until 2026-10-30 18:33:02.579799937 +0000 UTC m=+2.843264879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle") pod "router-default-56f55855fc-ctkk9" (UID: "3bc815a3-5bd4-4ec0-b417-29f36198d1be") : configmap references non-existent config key: service-ca.crt
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.579965    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls podName:db30aca3-472b-466d-be56-a6f8eb20b36a nodeName:}" failed. No retries permitted until 2026-10-30 18:33:02.579948683 +0000 UTC m=+2.843413608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-sqtt8" (UID: "db30aca3-472b-466d-be56-a6f8eb20b36a") : secret "samples-operator-tls" not found
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.580005    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:02.579996663 +0000 UTC m=+2.843461590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:01.580127 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:01.580033    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:02.580013742 +0000 UTC m=+2.843478668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:33:01.610060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.609993    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:33:01.613255 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:01.613221    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde844348_2c71_4773_b997_e1a79da8e4e1.slice/crio-235c103d72cb2df6ded5b1d135420a945a59eb4a58bfc0160c938e806fe20af1 WatchSource:0}: Error finding container 235c103d72cb2df6ded5b1d135420a945a59eb4a58bfc0160c938e806fe20af1: Status 404 returned error can't find the container with id 235c103d72cb2df6ded5b1d135420a945a59eb4a58bfc0160c938e806fe20af1
Oct 30 18:33:01.951100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.950999    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" event={"ID":"4c39f772-3455-4174-a1b8-49b8a5f74b30","Type":"ContainerStarted","Data":"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"}
Oct 30 18:33:01.951100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.951055    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" event={"ID":"4c39f772-3455-4174-a1b8-49b8a5f74b30","Type":"ContainerStarted","Data":"1cd8e2a65f96297698aaa51a63122100887f65384a316237371e8206e91c64a4"}
Oct 30 18:33:01.952483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.952455    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct" event={"ID":"263e2d58-6476-493e-9de0-319a8ed0c5f7","Type":"ContainerStarted","Data":"9632f0fda9e3c87831236876cdc13d691b6cba58144c35592424d9165c5ca284"}
Oct 30 18:33:01.953646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.953626    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5n4z" event={"ID":"db036c26-2a01-431b-a135-d3098bd05427","Type":"ContainerStarted","Data":"e7632eea07cf9a125af93393e005a93c8ba936c9cee7bdef5fd061772f794c28"}
Oct 30 18:33:01.955071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.955041    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8bh7" event={"ID":"8391c80c-4407-4fc4-9053-b336be68476c","Type":"ContainerStarted","Data":"da9b359683d86e2ae317117f23ae0329f98ffaeba66a63d0a795b54fbb6f2572"}
Oct 30 18:33:01.957220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.957205    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:33:01.957293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.957285    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd" event={"ID":"0fab5545-eaa2-487e-890b-f21a689ee08c","Type":"ContainerStarted","Data":"c18347146441df3747713cacce7adfee2bee65ce259f11ef86a7fb4a5d630225"}
Oct 30 18:33:01.959127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.959079    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" event={"ID":"de844348-2c71-4773-b997-e1a79da8e4e1","Type":"ContainerStarted","Data":"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"}
Oct 30 18:33:01.959243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.959152    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" event={"ID":"de844348-2c71-4773-b997-e1a79da8e4e1","Type":"ContainerStarted","Data":"235c103d72cb2df6ded5b1d135420a945a59eb4a58bfc0160c938e806fe20af1"}
Oct 30 18:33:01.983057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:01.983027    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:02.589690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.589653    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:02.589919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.589702    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:02.589919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.589737    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:02.589919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.589796    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:02.590054 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:02.589949    5065 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:33:02.590054 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:02.590018    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.589998836 +0000 UTC m=+4.853463784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:33:02.590420 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:02.590402    5065 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:02.590481 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:02.590448    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:04.59043794 +0000 UTC m=+4.853902879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:02.591140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.591121    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc815a3-5bd4-4ec0-b417-29f36198d1be-service-ca-bundle\") pod \"router-default-56f55855fc-ctkk9\" (UID: \"3bc815a3-5bd4-4ec0-b417-29f36198d1be\") " pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:02.594272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.594209    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/db30aca3-472b-466d-be56-a6f8eb20b36a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-sqtt8\" (UID: \"db30aca3-472b-466d-be56-a6f8eb20b36a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:02.645047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.645017    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"
Oct 30 18:33:02.648246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.648214    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:02.966152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.966126    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56f55855fc-ctkk9"]
Oct 30 18:33:02.971996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.971973    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:02.972587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.972554    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5n4z" event={"ID":"db036c26-2a01-431b-a135-d3098bd05427","Type":"ContainerStarted","Data":"3cc8261f85734e41935b9763fe8684e79b97f91663fb8ca9eae4489d2aa81598"}
Oct 30 18:33:02.997689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:02.997660    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8"]
Oct 30 18:33:03.383817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.383788    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:03.384034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.384014    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:03.667833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.667229    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:33:03.699889 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.699470    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nhsc4"
Oct 30 18:33:03.989142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.989080    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56f55855fc-ctkk9" event={"ID":"3bc815a3-5bd4-4ec0-b417-29f36198d1be","Type":"ContainerStarted","Data":"91f1d2ab7c150edd0bd2cdcaa4e39931187386bb3f5f9774149defa0b0d4f621"}
Oct 30 18:33:03.989609 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.989163    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56f55855fc-ctkk9" event={"ID":"3bc815a3-5bd4-4ec0-b417-29f36198d1be","Type":"ContainerStarted","Data":"b28145c853af3abcdea1197ed057efba869eef221fb918b142b73bbdab7970c1"}
Oct 30 18:33:03.995233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.995200    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8" event={"ID":"db30aca3-472b-466d-be56-a6f8eb20b36a","Type":"ContainerStarted","Data":"69fe00c0dd08e2dc869338583150b1c7deb3a06052e45776543bf112d73cb1be"}
Oct 30 18:33:03.998256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:03.997874    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lf2ct" event={"ID":"263e2d58-6476-493e-9de0-319a8ed0c5f7","Type":"ContainerStarted","Data":"2d289cbc7ca0c2c5d604548939713550e6727fc1e03113c8ae5e3ea47a728ec6"}
Oct 30 18:33:04.001724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.001699    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5n4z" event={"ID":"db036c26-2a01-431b-a135-d3098bd05427","Type":"ContainerStarted","Data":"8e9f8b78723234edc0be8ab9a31810441eab3c17a509ad45958236c508b58d12"}
Oct 30 18:33:04.005991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.005971    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8bh7" event={"ID":"8391c80c-4407-4fc4-9053-b336be68476c","Type":"ContainerStarted","Data":"4553462913d13385e37f64dcfcc2b6465726fbdb6850696ed188b8dffb596b46"}
Oct 30 18:33:04.613402 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.613358    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:04.613565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.613494    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:04.613968 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:04.613657    5065 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:33:04.613968 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:04.613743    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert podName:b6d938f8-d5bf-42a9-b4c5-169b0f0d6777 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:08.613716658 +0000 UTC m=+8.877181601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gjzjs" (UID: "b6d938f8-d5bf-42a9-b4c5-169b0f0d6777") : secret "networking-console-plugin-cert" not found
Oct 30 18:33:04.613968 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:04.613824    5065 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:04.613968 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:04.613857    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls podName:f03be630-d162-45d2-8d86-6dceae239940 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:08.613846247 +0000 UTC m=+8.877311187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvfzd" (UID: "f03be630-d162-45d2-8d86-6dceae239940") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:33:04.648962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.648926    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:04.652182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:04.652157    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:05.013873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.013842    5065 generic.go:356] "Generic (PLEG): container finished" podID="3b50a449-623e-44d0-bdb9-541e633dab54" containerID="e18a8a8a133dd03cda366ecf6ddec4638b023e5f9b78a0ca59fec68dd1856414" exitCode=0
Oct 30 18:33:05.014315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.013908    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ghhlw" event={"ID":"3b50a449-623e-44d0-bdb9-541e633dab54","Type":"ContainerDied","Data":"e18a8a8a133dd03cda366ecf6ddec4638b023e5f9b78a0ca59fec68dd1856414"}
Oct 30 18:33:05.014315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.014259    5065 scope.go:122] "RemoveContainer" containerID="e18a8a8a133dd03cda366ecf6ddec4638b023e5f9b78a0ca59fec68dd1856414"
Oct 30 18:33:05.015872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.015851    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8" event={"ID":"db30aca3-472b-466d-be56-a6f8eb20b36a","Type":"ContainerStarted","Data":"d89bd72f5604ee63e47e3e95ba5b82c49a7f6115106c087dae132de629012f45"}
Oct 30 18:33:05.015976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.015892    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-sqtt8" event={"ID":"db30aca3-472b-466d-be56-a6f8eb20b36a","Type":"ContainerStarted","Data":"36bd8073a2479be0bbfcaaa9afac48cd1f0d0fea007226bd986f6df622e3c363"}
Oct 30 18:33:05.017405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.017387    5065 generic.go:356] "Generic (PLEG): container finished" podID="822c5b89-fcb5-4102-bf57-9ca8e15b7c40" containerID="bd1cd3697f1648efbe1008a98c276b0c3c066c82f8de035be0f3050b9cf62512" exitCode=0
Oct 30 18:33:05.017482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.017462    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj" event={"ID":"822c5b89-fcb5-4102-bf57-9ca8e15b7c40","Type":"ContainerDied","Data":"bd1cd3697f1648efbe1008a98c276b0c3c066c82f8de035be0f3050b9cf62512"}
Oct 30 18:33:05.017736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.017712    5065 scope.go:122] "RemoveContainer" containerID="bd1cd3697f1648efbe1008a98c276b0c3c066c82f8de035be0f3050b9cf62512"
Oct 30 18:33:05.019422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.019403    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8bh7" event={"ID":"8391c80c-4407-4fc4-9053-b336be68476c","Type":"ContainerStarted","Data":"ccd917839bc9e7a4df73a033ac9a306ebe21dc72196c944bce92da2a6d54d1c3"}
Oct 30 18:33:05.019532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.019453    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:05.020991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.020968    5065 generic.go:356] "Generic (PLEG): container finished" podID="8c676795-0942-4bfc-8abd-65ab1071d237" containerID="55001668c689b135b3a878a507c25b28b1e688bb8a90cdf2aed20b4ff8d093e5" exitCode=0
Oct 30 18:33:05.021076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.021057    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc" event={"ID":"8c676795-0942-4bfc-8abd-65ab1071d237","Type":"ContainerDied","Data":"55001668c689b135b3a878a507c25b28b1e688bb8a90cdf2aed20b4ff8d093e5"}
Oct 30 18:33:05.021681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:05.021662    5065 scope.go:122] "RemoveContainer" containerID="55001668c689b135b3a878a507c25b28b1e688bb8a90cdf2aed20b4ff8d093e5"
Oct 30 18:33:06.026873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.026842    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ghhlw" event={"ID":"3b50a449-623e-44d0-bdb9-541e633dab54","Type":"ContainerStarted","Data":"517ad465524c4b084eec6907aa7679ec2ee61c4fa0ab5327897f5f77ee3c2dc0"}
Oct 30 18:33:06.028815 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.028787    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6klqj" event={"ID":"822c5b89-fcb5-4102-bf57-9ca8e15b7c40","Type":"ContainerStarted","Data":"9668a5c0d50b01b8ced269a456c1280789094f300ed0c58c665c202c2235f7e0"}
Oct 30 18:33:06.028930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.028844    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:06.028930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.028925    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:06.030866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.030844    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-llkhc" event={"ID":"8c676795-0942-4bfc-8abd-65ab1071d237","Type":"ContainerStarted","Data":"89c21105de69a79f7adff8f4f19363d984f2eac765237af776971f0eb46e9fce"}
Oct 30 18:33:06.031678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.031605    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:06.035642 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:06.035326    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rsttd"
Oct 30 18:33:08.382892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.382857    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:08.643016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.642939    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:08.643016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.642990    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:08.645574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.645551    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03be630-d162-45d2-8d86-6dceae239940-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvfzd\" (UID: \"f03be630-d162-45d2-8d86-6dceae239940\") " pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:08.645670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.645649    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d938f8-d5bf-42a9-b4c5-169b0f0d6777-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gjzjs\" (UID: \"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:08.648493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.648479    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"
Oct 30 18:33:08.661472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.661451    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvfzd"
Oct 30 18:33:08.925686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.925653    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gjzjs"]
Oct 30 18:33:08.927757 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:08.927720    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d938f8_d5bf_42a9_b4c5_169b0f0d6777.slice/crio-9e3b9c9784ff23488f86357ac85f70b32c9ca5cab22869cf700f2e0f5b3b0124 WatchSource:0}: Error finding container 9e3b9c9784ff23488f86357ac85f70b32c9ca5cab22869cf700f2e0f5b3b0124: Status 404 returned error can't find the container with id 9e3b9c9784ff23488f86357ac85f70b32c9ca5cab22869cf700f2e0f5b3b0124
Oct 30 18:33:08.948792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:08.948771    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvfzd"]
Oct 30 18:33:08.951121 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:08.951098    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03be630_d162_45d2_8d86_6dceae239940.slice/crio-f221e53eefafb625f218b5ba63c3fced95123c6e1a1145066eac05d82cea25af WatchSource:0}: Error finding container f221e53eefafb625f218b5ba63c3fced95123c6e1a1145066eac05d82cea25af: Status 404 returned error can't find the container with id f221e53eefafb625f218b5ba63c3fced95123c6e1a1145066eac05d82cea25af
Oct 30 18:33:09.044618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:09.044591    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs" event={"ID":"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777","Type":"ContainerStarted","Data":"9e3b9c9784ff23488f86357ac85f70b32c9ca5cab22869cf700f2e0f5b3b0124"}
Oct 30 18:33:09.046133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:09.046110    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvfzd" event={"ID":"f03be630-d162-45d2-8d86-6dceae239940","Type":"ContainerStarted","Data":"c56bad2249a6cbd49b6a8169973bc92167aab2391ff39f95b0731218f4222cb1"}
Oct 30 18:33:09.046233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:09.046151    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvfzd" event={"ID":"f03be630-d162-45d2-8d86-6dceae239940","Type":"ContainerStarted","Data":"f221e53eefafb625f218b5ba63c3fced95123c6e1a1145066eac05d82cea25af"}
Oct 30 18:33:10.051377 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:10.051342    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvfzd" event={"ID":"f03be630-d162-45d2-8d86-6dceae239940","Type":"ContainerStarted","Data":"8b0b1dd5ea1960b7d89930ef36c6732d2854ba977679340a230c710efbe26671"}
Oct 30 18:33:10.052747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:10.052724    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-gjzjs" event={"ID":"b6d938f8-d5bf-42a9-b4c5-169b0f0d6777","Type":"ContainerStarted","Data":"e2fd8a7798d8ad4530586f43ebb62d2d527b3542dc252b7b23476d30e555e8f8"}
Oct 30 18:33:10.951885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:10.951849    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:10.952055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:10.951993    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:10.953378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:10.953353    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56f55855fc-ctkk9"
Oct 30 18:33:12.065566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:12.065527    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvfzd" event={"ID":"f03be630-d162-45d2-8d86-6dceae239940","Type":"ContainerStarted","Data":"9205301ad1eeab65d8ed86fa3bf621005729fd90d7beefe3c4d9511acb497a02"}
Oct 30 18:33:15.033574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:15.033544    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l8bh7"
Oct 30 18:33:15.354169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:15.354079    5065 patch_prober.go:29] interesting pod/image-registry-5b594796db-ldb2p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:33:15.354169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:15.354152    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:33:17.427922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:17.427887    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:33:17.432914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:17.432884    5065 patch_prober.go:29] interesting pod/image-registry-5b594796db-ldb2p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:33:17.433052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:17.432939    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:33:19.004406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:19.004360    5065 patch_prober.go:29] interesting pod/image-registry-59c4bf777f-scwmb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:33:19.004841 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:19.004434    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:33:23.056172 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:23.056129    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:33:24.010519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:24.010484    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:33:24.802508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:24.802474    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-gjzjs_b6d938f8-d5bf-42a9-b4c5-169b0f0d6777/networking-console-plugin/0.log"
Oct 30 18:33:24.888102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:24.888059    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lr9k"
Oct 30 18:33:25.002829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:25.002801    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:33:25.204331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:25.204261    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/2.log"
Oct 30 18:33:26.402891 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.402851    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8bh7_8391c80c-4407-4fc4-9053-b336be68476c/dns/0.log"
Oct 30 18:33:26.606163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.606129    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8bh7_8391c80c-4407-4fc4-9053-b336be68476c/kube-rbac-proxy/0.log"
Oct 30 18:33:26.880028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.880000    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9g4zt"]
Oct 30 18:33:26.922440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.922418    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:26.977156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.977136    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:33:26.977365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.977347    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hh248\""
Oct 30 18:33:26.977504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.977490    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:33:26.978356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.978340    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:33:26.978493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:26.978481    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:33:27.086384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086358    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-tls\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086387    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086410    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7stq\" (UniqueName: \"kubernetes.io/projected/c334df43-4158-434b-bd57-ff8119a576a8-kube-api-access-x7stq\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086439    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-textfile\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086570    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-wtmp\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086593    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-root\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086608    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-metrics-client-ca\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086668    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-sys\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.086743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.086726    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187310    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7stq\" (UniqueName: \"kubernetes.io/projected/c334df43-4158-434b-bd57-ff8119a576a8-kube-api-access-x7stq\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187369    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-textfile\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187410    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-wtmp\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187437    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-root\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187462    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-metrics-client-ca\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187496    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-sys\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187528    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-root\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187532    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187568    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-wtmp\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187587    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c334df43-4158-434b-bd57-ff8119a576a8-sys\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187631    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-tls\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.187812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187656    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.188062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.187826    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-textfile\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.188062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.188033    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-metrics-client-ca\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.188160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.188136    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.190256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.190235    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-tls\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.190384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.190266    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c334df43-4158-434b-bd57-ff8119a576a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.199980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.199959    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7stq\" (UniqueName: \"kubernetes.io/projected/c334df43-4158-434b-bd57-ff8119a576a8-kube-api-access-x7stq\") pod \"node-exporter-9g4zt\" (UID: \"c334df43-4158-434b-bd57-ff8119a576a8\") " pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.200858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.200839    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-prl4g_0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/dns-node-resolver/0.log"
Oct 30 18:33:27.273046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.273024    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9g4zt"
Oct 30 18:33:27.318643 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:33:27.318621    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc334df43_4158_434b_bd57_ff8119a576a8.slice/crio-5cf8428f8ea2012a824beef1c19572f8bb0b299a1d63821c6aa987326d3a554b WatchSource:0}: Error finding container 5cf8428f8ea2012a824beef1c19572f8bb0b299a1d63821c6aa987326d3a554b: Status 404 returned error can't find the container with id 5cf8428f8ea2012a824beef1c19572f8bb0b299a1d63821c6aa987326d3a554b
Oct 30 18:33:27.402054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.402032    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59c4bf777f-scwmb_de844348-2c71-4773-b997-e1a79da8e4e1/registry/0.log"
Oct 30 18:33:27.432555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.432538    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:27.608872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:27.608844    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b594796db-ldb2p_4c39f772-3455-4174-a1b8-49b8a5f74b30/registry/0.log"
Oct 30 18:33:28.133965 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:28.133936    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9g4zt" event={"ID":"c334df43-4158-434b-bd57-ff8119a576a8","Type":"ContainerStarted","Data":"5cf8428f8ea2012a824beef1c19572f8bb0b299a1d63821c6aa987326d3a554b"}
Oct 30 18:33:28.201414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:28.201385    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fqpv4_2fa09131-1c47-4d63-82d2-c5174c60fd49/node-ca/0.log"
Oct 30 18:33:28.605937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:28.605909    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f55855fc-ctkk9_3bc815a3-5bd4-4ec0-b417-29f36198d1be/router/0.log"
Oct 30 18:33:29.139616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:29.139552    5065 generic.go:356] "Generic (PLEG): container finished" podID="c334df43-4158-434b-bd57-ff8119a576a8" containerID="4ad849aaece0d4d9a65172ad2cd1c74e27803dd502d8b727c198a9ec16021f83" exitCode=0
Oct 30 18:33:29.139616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:29.139595    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9g4zt" event={"ID":"c334df43-4158-434b-bd57-ff8119a576a8","Type":"ContainerDied","Data":"4ad849aaece0d4d9a65172ad2cd1c74e27803dd502d8b727c198a9ec16021f83"}
Oct 30 18:33:29.209946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:29.209921    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrlwc_b264fe31-7f47-4667-9563-3ec98cf45e9b/serve-healthcheck-canary/0.log"
Oct 30 18:33:30.145374 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:30.145328    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9g4zt" event={"ID":"c334df43-4158-434b-bd57-ff8119a576a8","Type":"ContainerStarted","Data":"f755afe9118b5fab6f86b57de0b7ffcf33ed7ab7bafe399649013e4592ab70da"}
Oct 30 18:33:30.145735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:30.145414    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9g4zt" event={"ID":"c334df43-4158-434b-bd57-ff8119a576a8","Type":"ContainerStarted","Data":"7a34ad782c84abb12ed98356c119f18b9a639ebaf72021427e0bc77dfdb32cf0"}
Oct 30 18:33:30.172860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:30.172803    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9g4zt" podStartSLOduration=2.654517448 podStartE2EDuration="4.172789861s" totalImagesPullingTime="1.432476437s" totalInitContainerRuntime="85.795976ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:30.170344602 +0000 UTC m=+30.433809549" watchObservedRunningTime="2026-10-30 18:33:30.172789861 +0000 UTC m=+30.436254806"
Oct 30 18:33:33.670370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:33.670341    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w7prc"
Oct 30 18:33:39.427154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:39.427120    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:33:42.449492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.449372    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry" containerID="cri-o://5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b" gracePeriod=30
Oct 30 18:33:42.687216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.687190    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:42.800717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800691    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.800717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800720    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.800914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800741    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.800914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800787    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.800914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800816    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.800914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800859    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.801136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800928    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.801136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.800965    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") pod \"4c39f772-3455-4174-a1b8-49b8a5f74b30\" (UID: \"4c39f772-3455-4174-a1b8-49b8a5f74b30\") "
Oct 30 18:33:42.801252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.801207    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:42.801628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.801317    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:42.803587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.803532    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:33:42.803587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.803565    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:33:42.803788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.803763    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "kube-api-access-hwpcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:33:42.803992 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.803970    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:33:42.804110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.804064    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:33:42.810706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.810681    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted" pod "4c39f772-3455-4174-a1b8-49b8a5f74b30" (UID: "4c39f772-3455-4174-a1b8-49b8a5f74b30"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:33:42.901926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901900    5065 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-bound-sa-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.901926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901924    5065 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-trusted-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901936    5065 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-image-registry-private-configuration\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901947    5065 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-tls\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901957    5065 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c39f772-3455-4174-a1b8-49b8a5f74b30-ca-trust-extracted\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901966    5065 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c39f772-3455-4174-a1b8-49b8a5f74b30-installation-pull-secrets\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901977    5065 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c39f772-3455-4174-a1b8-49b8a5f74b30-registry-certificates\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:42.902069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:42.901985    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpcn\" (UniqueName: \"kubernetes.io/projected/4c39f772-3455-4174-a1b8-49b8a5f74b30-kube-api-access-hwpcn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:43.201161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.201068    5065 generic.go:356] "Generic (PLEG): container finished" podID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerID="5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b" exitCode=0
Oct 30 18:33:43.201161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.201146    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b594796db-ldb2p"
Oct 30 18:33:43.201308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.201144    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" event={"ID":"4c39f772-3455-4174-a1b8-49b8a5f74b30","Type":"ContainerDied","Data":"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"}
Oct 30 18:33:43.201308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.201262    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b594796db-ldb2p" event={"ID":"4c39f772-3455-4174-a1b8-49b8a5f74b30","Type":"ContainerDied","Data":"1cd8e2a65f96297698aaa51a63122100887f65384a316237371e8206e91c64a4"}
Oct 30 18:33:43.201308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.201291    5065 scope.go:122] "RemoveContainer" containerID="5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"
Oct 30 18:33:43.210480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.210461    5065 scope.go:122] "RemoveContainer" containerID="5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"
Oct 30 18:33:43.210753 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:43.210725    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b\": container with ID starting with 5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b not found: ID does not exist" containerID="5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"
Oct 30 18:33:43.210807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.210764    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b"} err="failed to get container status \"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b\": rpc error: code = NotFound desc = could not find container \"5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b\": container with ID starting with 5ca57d7c1930172d97e77bee4aa636ff896ec7a4c955b5a145eb2231d4c74d1b not found: ID does not exist"
Oct 30 18:33:43.226463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.226433    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:33:43.231535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.231511    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b594796db-ldb2p"]
Oct 30 18:33:43.886979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:43.886952    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" path="/var/lib/kubelet/pods/4c39f772-3455-4174-a1b8-49b8a5f74b30/volumes"
Oct 30 18:33:48.729052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:48.729021    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:33:48.729532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:48.729248    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-glkck-6hcwm" podUID="6a5210ed-818c-48af-9883-0919354e4ae3" containerName="insights-gathering" containerID="cri-o://98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c" gracePeriod=30
Oct 30 18:33:48.749931 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:48.749906    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5210ed_818c_48af_9883_0919354e4ae3.slice/crio-conmon-98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:33:48.965811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:48.965790    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:49.051153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051131    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") pod \"6a5210ed-818c-48af-9883-0919354e4ae3\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") "
Oct 30 18:33:49.051271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051171    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") pod \"6a5210ed-818c-48af-9883-0919354e4ae3\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") "
Oct 30 18:33:49.051271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051236    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") pod \"6a5210ed-818c-48af-9883-0919354e4ae3\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") "
Oct 30 18:33:49.051271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051263    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") pod \"6a5210ed-818c-48af-9883-0919354e4ae3\" (UID: \"6a5210ed-818c-48af-9883-0919354e4ae3\") "
Oct 30 18:33:49.051559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051534    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path" pod "6a5210ed-818c-48af-9883-0919354e4ae3" (UID: "6a5210ed-818c-48af-9883-0919354e4ae3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:33:49.051625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051587    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle" pod "6a5210ed-818c-48af-9883-0919354e4ae3" (UID: "6a5210ed-818c-48af-9883-0919354e4ae3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:49.051678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.051637    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle" pod "6a5210ed-818c-48af-9883-0919354e4ae3" (UID: "6a5210ed-818c-48af-9883-0919354e4ae3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:49.053639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.053618    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6" pod "6a5210ed-818c-48af-9883-0919354e4ae3" (UID: "6a5210ed-818c-48af-9883-0919354e4ae3"). InnerVolumeSpecName "kube-api-access-npxd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:33:49.152054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.152030    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxd6\" (UniqueName: \"kubernetes.io/projected/6a5210ed-818c-48af-9883-0919354e4ae3-kube-api-access-npxd6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:49.152054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.152052    5065 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-trusted-ca-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:49.152197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.152062    5065 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6a5210ed-818c-48af-9883-0919354e4ae3-archives-path\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:49.152197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.152071    5065 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5210ed-818c-48af-9883-0919354e4ae3-service-ca-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:49.232129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.232095    5065 generic.go:356] "Generic (PLEG): container finished" podID="6a5210ed-818c-48af-9883-0919354e4ae3" containerID="98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c" exitCode=2
Oct 30 18:33:49.232244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.232127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-glkck-6hcwm" event={"ID":"6a5210ed-818c-48af-9883-0919354e4ae3","Type":"ContainerDied","Data":"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"}
Oct 30 18:33:49.232244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.232162    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-glkck-6hcwm"
Oct 30 18:33:49.232244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.232176    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-glkck-6hcwm" event={"ID":"6a5210ed-818c-48af-9883-0919354e4ae3","Type":"ContainerDied","Data":"44108dd0d2df26fcf6a97390abba4d4e36126bf41811c6e02785d029322b3cfd"}
Oct 30 18:33:49.232244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.232198    5065 scope.go:122] "RemoveContainer" containerID="98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"
Oct 30 18:33:49.241618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.241603    5065 scope.go:122] "RemoveContainer" containerID="98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"
Oct 30 18:33:49.241877 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:49.241858    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c\": container with ID starting with 98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c not found: ID does not exist" containerID="98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"
Oct 30 18:33:49.241917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.241886    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c"} err="failed to get container status \"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c\": rpc error: code = NotFound desc = could not find container \"98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c\": container with ID starting with 98ace1d54cef845d4a35cd478063ff161b68f2e4e2dd66755d28cabda0d22b3c not found: ID does not exist"
Oct 30 18:33:49.255303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.255282    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:33:49.260437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.260409    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-glkck-6hcwm"]
Oct 30 18:33:49.887384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:33:49.887353    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5210ed-818c-48af-9883-0919354e4ae3" path="/var/lib/kubelet/pods/6a5210ed-818c-48af-9883-0919354e4ae3/volumes"
Oct 30 18:33:53.055757 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:33:53.055710    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:34:04.447042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.447003    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" containerName="registry" containerID="cri-o://11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9" gracePeriod=30
Oct 30 18:34:04.697017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.696991    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:34:04.864514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864475    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864533    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864571    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864593    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864626    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864664    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864710    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.864872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.864750    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") pod \"de844348-2c71-4773-b997-e1a79da8e4e1\" (UID: \"de844348-2c71-4773-b997-e1a79da8e4e1\") "
Oct 30 18:34:04.865273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.865218    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:04.865567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.865536    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:04.867425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.867378    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "kube-api-access-nhh8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:04.867605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.867573    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:04.867665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.867629    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:04.867705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.867694    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:04.867932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.867906    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:04.872499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.872465    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted" pod "de844348-2c71-4773-b997-e1a79da8e4e1" (UID: "de844348-2c71-4773-b997-e1a79da8e4e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:34:04.965829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965805    5065 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-installation-pull-secrets\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965829    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhh8j\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-kube-api-access-nhh8j\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965839    5065 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-trusted-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965850    5065 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de844348-2c71-4773-b997-e1a79da8e4e1-ca-trust-extracted\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965860    5065 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-registry-tls\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965868    5065 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de844348-2c71-4773-b997-e1a79da8e4e1-bound-sa-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965878    5065 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de844348-2c71-4773-b997-e1a79da8e4e1-image-registry-private-configuration\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:04.965954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:04.965888    5065 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de844348-2c71-4773-b997-e1a79da8e4e1-registry-certificates\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:05.307855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.307825    5065 generic.go:356] "Generic (PLEG): container finished" podID="de844348-2c71-4773-b997-e1a79da8e4e1" containerID="11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9" exitCode=0
Oct 30 18:34:05.307971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.307876    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb"
Oct 30 18:34:05.307971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.307913    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" event={"ID":"de844348-2c71-4773-b997-e1a79da8e4e1","Type":"ContainerDied","Data":"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"}
Oct 30 18:34:05.307971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.307969    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c4bf777f-scwmb" event={"ID":"de844348-2c71-4773-b997-e1a79da8e4e1","Type":"ContainerDied","Data":"235c103d72cb2df6ded5b1d135420a945a59eb4a58bfc0160c938e806fe20af1"}
Oct 30 18:34:05.308119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.307997    5065 scope.go:122] "RemoveContainer" containerID="11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"
Oct 30 18:34:05.318585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.318563    5065 scope.go:122] "RemoveContainer" containerID="11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"
Oct 30 18:34:05.318885 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:34:05.318867    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9\": container with ID starting with 11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9 not found: ID does not exist" containerID="11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"
Oct 30 18:34:05.318954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.318896    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9"} err="failed to get container status \"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9\": rpc error: code = NotFound desc = could not find container \"11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9\": container with ID starting with 11299a3a67e78efd3ae94fcdff5838aa2e1f1cae1cb819dc9af663440341f9d9 not found: ID does not exist"
Oct 30 18:34:05.333616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.333594    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:34:05.338038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.338019    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59c4bf777f-scwmb"]
Oct 30 18:34:05.887589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:34:05.887556    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" path="/var/lib/kubelet/pods/de844348-2c71-4773-b997-e1a79da8e4e1/volumes"
Oct 30 18:34:23.057882 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:34:23.057844    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:34:53.055293 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:34:53.055251    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:35:23.059152 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:35:23.059039    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:35:53.057676 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:35:53.057638    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:36:23.056425 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:36:23.056387    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:36:53.056467 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:36:53.056425    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:37:01.339454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339422    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"]
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339740    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6a5210ed-818c-48af-9883-0919354e4ae3" containerName="insights-gathering"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339752    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5210ed-818c-48af-9883-0919354e4ae3" containerName="insights-gathering"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339761    5065 cpu_manager.go:414] "containerMap: removing container" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" containerName="registry"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339766    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" containerName="registry"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339786    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339791    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339838    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c39f772-3455-4174-a1b8-49b8a5f74b30" containerName="registry"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339848    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5210ed-818c-48af-9883-0919354e4ae3" containerName="insights-gathering"
Oct 30 18:37:01.339906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.339854    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="de844348-2c71-4773-b997-e1a79da8e4e1" containerName="registry"
Oct 30 18:37:01.342975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.342957    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.350462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.350441    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:37:01.350558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.350460    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:37:01.354652 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.354629    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-tbhdz\""
Oct 30 18:37:01.354873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.354827    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"]
Oct 30 18:37:01.466143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.466122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.466257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.466149    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.466257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.466200    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.567041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.567015    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.567179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.567081    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.567179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.567125    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.567455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.567438    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.567510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.567466    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.584359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.584336    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.659125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.659052    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:01.934565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:01.934524    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"]
Oct 30 18:37:01.937305 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:01.937277    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b91b10_947a_476b_a5a0_cd1cedf8ebc3.slice/crio-e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62 WatchSource:0}: Error finding container e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62: Status 404 returned error can't find the container with id e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62
Oct 30 18:37:02.022914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:02.022878    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p" event={"ID":"83b91b10-947a-476b-a5a0-cd1cedf8ebc3","Type":"ContainerStarted","Data":"e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62"}
Oct 30 18:37:04.148779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.148743    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"]
Oct 30 18:37:04.153118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.153079    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.166526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.166502    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"]
Oct 30 18:37:04.167286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.167252    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xlrf4\""
Oct 30 18:37:04.255454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.255421    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"]
Oct 30 18:37:04.258834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.258814    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.271013 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.270992    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-vhqjn\""
Oct 30 18:37:04.274065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.274039    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"]
Oct 30 18:37:04.290759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.290731    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.290884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.290831    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6xc\" (UniqueName: \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.291136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.290911    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.356739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.356713    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"]
Oct 30 18:37:04.360348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.360329    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.365755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.365731    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mv7c4\""
Oct 30 18:37:04.372617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.372589    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"]
Oct 30 18:37:04.391976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.391948    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.392125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392006    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6xc\" (UniqueName: \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.392125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392045    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbzs\" (UniqueName: \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.392125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392074    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.392125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392107    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.392125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392123    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.392387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392367    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.392453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.392405    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.407192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.407131    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6xc\" (UniqueName: \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.442501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.442470    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"]
Oct 30 18:37:04.446240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.446220    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.456930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.456910    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"]
Oct 30 18:37:04.466677 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.466659    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bzjmt\""
Oct 30 18:37:04.471796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.471776    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:04.492937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.492836    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.494071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494047    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r79q\" (UniqueName: \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.494189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494114    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbzs\" (UniqueName: \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.494189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494143    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.494189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494170    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.494318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494198    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.494318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494224    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.494318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494257    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmn8\" (UniqueName: \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.494318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494298    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.494800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494776    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.494902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.494860    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.508785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.508765    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbzs\" (UniqueName: \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.580641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.580612    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:04.595123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595075    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.595238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595148    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r79q\" (UniqueName: \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.595238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595183    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.595238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595218    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.595407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595250    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmn8\" (UniqueName: \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.595407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595290    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.595532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595417    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.595903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595630    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.595903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595740    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.596051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.595910    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.609615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.609596    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r79q\" (UniqueName: \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.609711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.609689    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmn8\" (UniqueName: \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.678270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.678200    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:04.758866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.758839    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"]
Oct 30 18:37:04.762696 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:04.762668    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd122dd8_3aec_407b_9407_7039473477f4.slice/crio-1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4 WatchSource:0}: Error finding container 1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4: Status 404 returned error can't find the container with id 1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4
Oct 30 18:37:04.765385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.765363    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:04.886646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.886619    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"]
Oct 30 18:37:04.890701 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:04.890676    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbc60c3_599a_4fde_a5dd_fc899932af6a.slice/crio-6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb WatchSource:0}: Error finding container 6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb: Status 404 returned error can't find the container with id 6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb
Oct 30 18:37:04.969218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:04.969040    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"]
Oct 30 18:37:04.971413 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:04.971388    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce36f61_1140_4393_a689_af8264e97b43.slice/crio-710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860 WatchSource:0}: Error finding container 710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860: Status 404 returned error can't find the container with id 710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860
Oct 30 18:37:05.037379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.037355    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5" event={"ID":"3ce36f61-1140-4393-a689-af8264e97b43","Type":"ContainerStarted","Data":"710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860"}
Oct 30 18:37:05.038784 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.038755    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4" event={"ID":"bbbc60c3-599a-4fde-a5dd-fc899932af6a","Type":"ContainerStarted","Data":"6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb"}
Oct 30 18:37:05.040135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.040109    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerStarted","Data":"1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4"}
Oct 30 18:37:05.058827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.058808    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"]
Oct 30 18:37:05.060708 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:05.060685    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a5bc43_7467_4d2e_b234_b7460228b455.slice/crio-6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496 WatchSource:0}: Error finding container 6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496: Status 404 returned error can't find the container with id 6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496
Oct 30 18:37:05.169684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.169652    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"]
Oct 30 18:37:05.174911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.174889    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.180143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.180120    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sz2z9\""
Oct 30 18:37:05.196414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.196382    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"]
Oct 30 18:37:05.301794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.301754    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.301949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.301839    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.301949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.301866    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtzg\" (UniqueName: \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.403626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.402507    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.403626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.402597    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.403626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.402630    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtzg\" (UniqueName: \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.403626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.403342    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.403626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.403578    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.423505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.423462    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtzg\" (UniqueName: \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.497244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.496983    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:05.522917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.522892    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:37:05.526489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.526463    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.555541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.555253    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n2l29\""
Oct 30 18:37:05.558193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.556984    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:37:05.576917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.576893    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606461    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c77l\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606563    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606595    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606640    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606689    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606733    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606757    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.607757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.606784    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707564    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707625    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c77l\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707669    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707702    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707742    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707774    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707816    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707844    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.707874    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.709390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.708751    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.711767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.710438    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.711767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.711432    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.711767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.711674    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.712327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.712279    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.715785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.715759    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.719251 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.719225    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.731181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.731134    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.736697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.736654    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c77l\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\") pod \"maas-default-gateway-openshift-default-855f9574cf-mz99s\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:05.832222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.832055    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"]
Oct 30 18:37:05.838526 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:05.838493    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0379437a_e747_4eb0_b1e0_176d617c36c2.slice/crio-a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf WatchSource:0}: Error finding container a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf: Status 404 returned error can't find the container with id a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf
Oct 30 18:37:05.848283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:05.848263    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:06.051507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:06.051396    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g" event={"ID":"0379437a-e747-4eb0-b1e0-176d617c36c2","Type":"ContainerStarted","Data":"a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf"}
Oct 30 18:37:06.053346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:06.053316    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq" event={"ID":"d1a5bc43-7467-4d2e-b234-b7460228b455","Type":"ContainerStarted","Data":"6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496"}
Oct 30 18:37:06.155548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:06.155468    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:37:06.159556 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:06.159522    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9c71a8_8cf5_44d2_b254_3d27b08b7f95.slice/crio-84342a9c4ea28e4b29ea2dc69d561a27979b189cea02ffbc74b46844b197e84f WatchSource:0}: Error finding container 84342a9c4ea28e4b29ea2dc69d561a27979b189cea02ffbc74b46844b197e84f: Status 404 returned error can't find the container with id 84342a9c4ea28e4b29ea2dc69d561a27979b189cea02ffbc74b46844b197e84f
Oct 30 18:37:07.065215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:07.065129    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerStarted","Data":"84342a9c4ea28e4b29ea2dc69d561a27979b189cea02ffbc74b46844b197e84f"}
Oct 30 18:37:12.130776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:12.130740    5065 generic.go:356] "Generic (PLEG): container finished" podID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerID="674ca1607e3505444e720144ef114380318144969aaf85aa7aa00fd93edda3df" exitCode=0
Oct 30 18:37:12.131338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:12.130809    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p" event={"ID":"83b91b10-947a-476b-a5a0-cd1cedf8ebc3","Type":"ContainerDied","Data":"674ca1607e3505444e720144ef114380318144969aaf85aa7aa00fd93edda3df"}
Oct 30 18:37:12.659029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:12.658556    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:37:12.659029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:12.658642    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:37:12.659029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:12.658687    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:37:13.143134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:13.139606    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerStarted","Data":"bd14c6effb23e59882d00a5f59b151664f0808422bc3fc22caae214d783968a8"}
Oct 30 18:37:13.143134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:13.143004    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerStarted","Data":"4d01a3153dd6e2f2e2e64c02dbc6bbbaaca0a1c770e970780841dd279bf93494"}
Oct 30 18:37:13.849164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:13.849098    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:13.854235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:13.854212    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:13.896901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:13.896852    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podStartSLOduration=2.402691915 podStartE2EDuration="8.896838562s" totalImagesPullingTime="6.494146647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:13.217328886 +0000 UTC m=+253.480793853" watchObservedRunningTime="2026-10-30 18:37:13.896838562 +0000 UTC m=+254.160303506"
Oct 30 18:37:14.149372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.149289    5065 generic.go:356] "Generic (PLEG): container finished" podID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerID="53bb94ef6dde8792dec0030a96b7da075473c21a835edddfff7ceac7259c2cb5" exitCode=0
Oct 30 18:37:14.149844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.149394    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g" event={"ID":"0379437a-e747-4eb0-b1e0-176d617c36c2","Type":"ContainerDied","Data":"53bb94ef6dde8792dec0030a96b7da075473c21a835edddfff7ceac7259c2cb5"}
Oct 30 18:37:14.152276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.152250    5065 generic.go:356] "Generic (PLEG): container finished" podID="3ce36f61-1140-4393-a689-af8264e97b43" containerID="63c45231628a16ff35b8c5aedd28b14c354fb50ae6cf01b28ed90aed4520b9e3" exitCode=0
Oct 30 18:37:14.152422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.152331    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5" event={"ID":"3ce36f61-1140-4393-a689-af8264e97b43","Type":"ContainerDied","Data":"63c45231628a16ff35b8c5aedd28b14c354fb50ae6cf01b28ed90aed4520b9e3"}
Oct 30 18:37:14.154612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.154041    5065 generic.go:356] "Generic (PLEG): container finished" podID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerID="62f6897992c06193f2494c81c2f6782370b994901811d468aa49c1f3470a7053" exitCode=0
Oct 30 18:37:14.154612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.154146    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4" event={"ID":"bbbc60c3-599a-4fde-a5dd-fc899932af6a","Type":"ContainerDied","Data":"62f6897992c06193f2494c81c2f6782370b994901811d468aa49c1f3470a7053"}
Oct 30 18:37:14.155958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.155929    5065 generic.go:356] "Generic (PLEG): container finished" podID="cd122dd8-3aec-407b-9407-7039473477f4" containerID="bd14c6effb23e59882d00a5f59b151664f0808422bc3fc22caae214d783968a8" exitCode=0
Oct 30 18:37:14.156056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.156032    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerDied","Data":"bd14c6effb23e59882d00a5f59b151664f0808422bc3fc22caae214d783968a8"}
Oct 30 18:37:14.158476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.158451    5065 generic.go:356] "Generic (PLEG): container finished" podID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerID="75d186e3b1c5710b8b45f00a258f05769bc535320d2b0124567ca6428548140f" exitCode=0
Oct 30 18:37:14.158657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.158636    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq" event={"ID":"d1a5bc43-7467-4d2e-b234-b7460228b455","Type":"ContainerDied","Data":"75d186e3b1c5710b8b45f00a258f05769bc535320d2b0124567ca6428548140f"}
Oct 30 18:37:14.159133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.159114    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:14.159971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:14.159953    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:37:15.164476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:15.164445    5065 generic.go:356] "Generic (PLEG): container finished" podID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerID="280814c4f9d47e004cfc5b42bc3c8d5caa69e796b2907c736910ebc293cae373" exitCode=0
Oct 30 18:37:15.164940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:15.164533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p" event={"ID":"83b91b10-947a-476b-a5a0-cd1cedf8ebc3","Type":"ContainerDied","Data":"280814c4f9d47e004cfc5b42bc3c8d5caa69e796b2907c736910ebc293cae373"}
Oct 30 18:37:15.166527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:15.166503    5065 generic.go:356] "Generic (PLEG): container finished" podID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerID="164330c26d34d63a254f2250f1be595f0f4d8c24eec320469078c4009a62f975" exitCode=0
Oct 30 18:37:15.166621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:15.166588    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g" event={"ID":"0379437a-e747-4eb0-b1e0-176d617c36c2","Type":"ContainerDied","Data":"164330c26d34d63a254f2250f1be595f0f4d8c24eec320469078c4009a62f975"}
Oct 30 18:37:16.173860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.173829    5065 generic.go:356] "Generic (PLEG): container finished" podID="3ce36f61-1140-4393-a689-af8264e97b43" containerID="be21fc61b11dac555c7db2c1197ce5da24e890de305116ed502770901368fbd0" exitCode=0
Oct 30 18:37:16.174372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.173864    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5" event={"ID":"3ce36f61-1140-4393-a689-af8264e97b43","Type":"ContainerDied","Data":"be21fc61b11dac555c7db2c1197ce5da24e890de305116ed502770901368fbd0"}
Oct 30 18:37:16.178253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.178229    5065 generic.go:356] "Generic (PLEG): container finished" podID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerID="0350f0eff974b099c1dc087e70d89b08f16148191bcb8412057cdf28b8633ac6" exitCode=0
Oct 30 18:37:16.178354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.178319    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4" event={"ID":"bbbc60c3-599a-4fde-a5dd-fc899932af6a","Type":"ContainerDied","Data":"0350f0eff974b099c1dc087e70d89b08f16148191bcb8412057cdf28b8633ac6"}
Oct 30 18:37:16.182356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.182179    5065 generic.go:356] "Generic (PLEG): container finished" podID="cd122dd8-3aec-407b-9407-7039473477f4" containerID="0fff496dcf7f44ae2d4c14a165c630056061ff1a45b4696d71fc3f71671f6b15" exitCode=0
Oct 30 18:37:16.182356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.182260    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerDied","Data":"0fff496dcf7f44ae2d4c14a165c630056061ff1a45b4696d71fc3f71671f6b15"}
Oct 30 18:37:16.184738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.184710    5065 generic.go:356] "Generic (PLEG): container finished" podID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerID="b5f36e364f1ba4deaf25c9146cdba69af733fd832e8196d9972eb26b7f2e7f52" exitCode=0
Oct 30 18:37:16.185177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:16.185151    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq" event={"ID":"d1a5bc43-7467-4d2e-b234-b7460228b455","Type":"ContainerDied","Data":"b5f36e364f1ba4deaf25c9146cdba69af733fd832e8196d9972eb26b7f2e7f52"}
Oct 30 18:37:23.087206 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:37:23.087069    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:37:28.300613 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:28.300573    5065 generic.go:356] "Generic (PLEG): container finished" podID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerID="f3bdc49ed183e9b0e2515c157fd13588911ded28e1591a1646f4809e9fbb1846" exitCode=0
Oct 30 18:37:28.301140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:28.300663    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p" event={"ID":"83b91b10-947a-476b-a5a0-cd1cedf8ebc3","Type":"ContainerDied","Data":"f3bdc49ed183e9b0e2515c157fd13588911ded28e1591a1646f4809e9fbb1846"}
Oct 30 18:37:28.305139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:28.305067    5065 generic.go:356] "Generic (PLEG): container finished" podID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerID="32e0b8a9c749b9d3fb97315f3b19f1f58d2e3b4e6a1e3c51f127b9a01a4836c4" exitCode=0
Oct 30 18:37:28.305264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:28.305182    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g" event={"ID":"0379437a-e747-4eb0-b1e0-176d617c36c2","Type":"ContainerDied","Data":"32e0b8a9c749b9d3fb97315f3b19f1f58d2e3b4e6a1e3c51f127b9a01a4836c4"}
Oct 30 18:37:29.311979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.311955    5065 generic.go:356] "Generic (PLEG): container finished" podID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerID="fc1f4a4fbe0a1d5a569c0c102354fc8a24ccc3d59844e42e6d2667db595a4110" exitCode=0
Oct 30 18:37:29.312315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.312045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq" event={"ID":"d1a5bc43-7467-4d2e-b234-b7460228b455","Type":"ContainerDied","Data":"fc1f4a4fbe0a1d5a569c0c102354fc8a24ccc3d59844e42e6d2667db595a4110"}
Oct 30 18:37:29.502816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.502792    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:29.517127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.517075    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:29.637530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637499    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\") pod \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") "
Oct 30 18:37:29.637695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637588    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\") pod \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") "
Oct 30 18:37:29.637695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637629    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\" (UniqueName: \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\") pod \"0379437a-e747-4eb0-b1e0-176d617c36c2\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") "
Oct 30 18:37:29.637695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637675    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\") pod \"0379437a-e747-4eb0-b1e0-176d617c36c2\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") "
Oct 30 18:37:29.637854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637721    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\") pod \"0379437a-e747-4eb0-b1e0-176d617c36c2\" (UID: \"0379437a-e747-4eb0-b1e0-176d617c36c2\") "
Oct 30 18:37:29.637854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.637764    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\") pod \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\" (UID: \"83b91b10-947a-476b-a5a0-cd1cedf8ebc3\") "
Oct 30 18:37:29.639544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.639511    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle" pod "0379437a-e747-4eb0-b1e0-176d617c36c2" (UID: "0379437a-e747-4eb0-b1e0-176d617c36c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:29.639672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.639511    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle" pod "83b91b10-947a-476b-a5a0-cd1cedf8ebc3" (UID: "83b91b10-947a-476b-a5a0-cd1cedf8ebc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:29.640698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.640670    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg" pod "0379437a-e747-4eb0-b1e0-176d617c36c2" (UID: "0379437a-e747-4eb0-b1e0-176d617c36c2"). InnerVolumeSpecName "kube-api-access-wrtzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:29.641141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.641117    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7" pod "83b91b10-947a-476b-a5a0-cd1cedf8ebc3" (UID: "83b91b10-947a-476b-a5a0-cd1cedf8ebc3"). InnerVolumeSpecName "kube-api-access-l9zg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:29.645177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.645140    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util" pod "0379437a-e747-4eb0-b1e0-176d617c36c2" (UID: "0379437a-e747-4eb0-b1e0-176d617c36c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:29.645649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.645623    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util" pod "83b91b10-947a-476b-a5a0-cd1cedf8ebc3" (UID: "83b91b10-947a-476b-a5a0-cd1cedf8ebc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:29.738833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738802    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:29.738833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738829    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:29.738967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738844    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtzg\" (UniqueName: \"kubernetes.io/projected/0379437a-e747-4eb0-b1e0-176d617c36c2-kube-api-access-wrtzg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:29.738967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738857    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:29.738967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738869    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0379437a-e747-4eb0-b1e0-176d617c36c2-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:29.738967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:29.738881    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/83b91b10-947a-476b-a5a0-cd1cedf8ebc3-kube-api-access-l9zg7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:30.318292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.318258    5065 generic.go:356] "Generic (PLEG): container finished" podID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerID="9f828d0101dba2b7e5a8a2aa118342caf4cc5ff41cae14351e723d619eee3d52" exitCode=0
Oct 30 18:37:30.318651 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.318335    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4" event={"ID":"bbbc60c3-599a-4fde-a5dd-fc899932af6a","Type":"ContainerDied","Data":"9f828d0101dba2b7e5a8a2aa118342caf4cc5ff41cae14351e723d619eee3d52"}
Oct 30 18:37:30.320587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.320561    5065 generic.go:356] "Generic (PLEG): container finished" podID="cd122dd8-3aec-407b-9407-7039473477f4" containerID="61ab3d8eb95840d31ce1700c61f18067135cafcd5cf67e8efe5b50d754357f4d" exitCode=0
Oct 30 18:37:30.320731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.320650    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerDied","Data":"61ab3d8eb95840d31ce1700c61f18067135cafcd5cf67e8efe5b50d754357f4d"}
Oct 30 18:37:30.322451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.322428    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p" event={"ID":"83b91b10-947a-476b-a5a0-cd1cedf8ebc3","Type":"ContainerDied","Data":"e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62"}
Oct 30 18:37:30.322451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.322438    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4jw6p"
Oct 30 18:37:30.322634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.322466    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9433a4200e74f016d1db09f13714634334302c4a6594069cf1b9f22c29a2d62"
Oct 30 18:37:30.324862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.324840    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g" event={"ID":"0379437a-e747-4eb0-b1e0-176d617c36c2","Type":"ContainerDied","Data":"a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf"}
Oct 30 18:37:30.324960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.324879    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a122aa409acb1b306af2113e7ff0afd9df943e35a7995f33355e347f669709bf"
Oct 30 18:37:30.324960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.324924    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cs26g"
Oct 30 18:37:30.327252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.327231    5065 generic.go:356] "Generic (PLEG): container finished" podID="3ce36f61-1140-4393-a689-af8264e97b43" containerID="a42dc66d63b893323994a28e81c90912a0ce726ceb17b228fd6224433847176d" exitCode=0
Oct 30 18:37:30.327341 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.327305    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5" event={"ID":"3ce36f61-1140-4393-a689-af8264e97b43","Type":"ContainerDied","Data":"a42dc66d63b893323994a28e81c90912a0ce726ceb17b228fd6224433847176d"}
Oct 30 18:37:30.477455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.477431    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:30.646403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.646339    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\" (UniqueName: \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\") pod \"d1a5bc43-7467-4d2e-b234-b7460228b455\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") "
Oct 30 18:37:30.646403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.646375    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\") pod \"d1a5bc43-7467-4d2e-b234-b7460228b455\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") "
Oct 30 18:37:30.646568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.646435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\") pod \"d1a5bc43-7467-4d2e-b234-b7460228b455\" (UID: \"d1a5bc43-7467-4d2e-b234-b7460228b455\") "
Oct 30 18:37:30.646878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.646854    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle" pod "d1a5bc43-7467-4d2e-b234-b7460228b455" (UID: "d1a5bc43-7467-4d2e-b234-b7460228b455"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:30.648672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.648644    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8" pod "d1a5bc43-7467-4d2e-b234-b7460228b455" (UID: "d1a5bc43-7467-4d2e-b234-b7460228b455"). InnerVolumeSpecName "kube-api-access-qrmn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:30.651754 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.651733    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util" pod "d1a5bc43-7467-4d2e-b234-b7460228b455" (UID: "d1a5bc43-7467-4d2e-b234-b7460228b455"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:30.747147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.747123    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrmn8\" (UniqueName: \"kubernetes.io/projected/d1a5bc43-7467-4d2e-b234-b7460228b455-kube-api-access-qrmn8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:30.747147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.747146    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:30.747272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:30.747155    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bc43-7467-4d2e-b234-b7460228b455-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.334292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.334241    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq" event={"ID":"d1a5bc43-7467-4d2e-b234-b7460228b455","Type":"ContainerDied","Data":"6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496"}
Oct 30 18:37:31.334292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.334284    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjhrq"
Oct 30 18:37:31.334725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.334312    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df286ab692aca418915361150eafec2bad0a559b735fdf28ba3f76d95f4d496"
Oct 30 18:37:31.501591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.501565    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:31.504117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.504080    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:31.507740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.507725    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:31.653617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653539    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\") pod \"cd122dd8-3aec-407b-9407-7039473477f4\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") "
Oct 30 18:37:31.653617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653581    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\") pod \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") "
Oct 30 18:37:31.653836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653621    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\") pod \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") "
Oct 30 18:37:31.653836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653667    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\" (UniqueName: \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\") pod \"cd122dd8-3aec-407b-9407-7039473477f4\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") "
Oct 30 18:37:31.653836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653755    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\" (UniqueName: \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\") pod \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\" (UID: \"bbbc60c3-599a-4fde-a5dd-fc899932af6a\") "
Oct 30 18:37:31.653836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653791    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\") pod \"3ce36f61-1140-4393-a689-af8264e97b43\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") "
Oct 30 18:37:31.653836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653822    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\" (UniqueName: \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\") pod \"3ce36f61-1140-4393-a689-af8264e97b43\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") "
Oct 30 18:37:31.654080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653849    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\") pod \"cd122dd8-3aec-407b-9407-7039473477f4\" (UID: \"cd122dd8-3aec-407b-9407-7039473477f4\") "
Oct 30 18:37:31.654080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.653956    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\") pod \"3ce36f61-1140-4393-a689-af8264e97b43\" (UID: \"3ce36f61-1140-4393-a689-af8264e97b43\") "
Oct 30 18:37:31.654331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.654305    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle" pod "bbbc60c3-599a-4fde-a5dd-fc899932af6a" (UID: "bbbc60c3-599a-4fde-a5dd-fc899932af6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.655016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.654989    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle" pod "cd122dd8-3aec-407b-9407-7039473477f4" (UID: "cd122dd8-3aec-407b-9407-7039473477f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.655148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.655130    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle" pod "3ce36f61-1140-4393-a689-af8264e97b43" (UID: "3ce36f61-1140-4393-a689-af8264e97b43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.656237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.656210    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs" pod "bbbc60c3-599a-4fde-a5dd-fc899932af6a" (UID: "bbbc60c3-599a-4fde-a5dd-fc899932af6a"). InnerVolumeSpecName "kube-api-access-qgbzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:31.656710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.656690    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc" pod "cd122dd8-3aec-407b-9407-7039473477f4" (UID: "cd122dd8-3aec-407b-9407-7039473477f4"). InnerVolumeSpecName "kube-api-access-9j6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:31.656710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.656690    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q" pod "3ce36f61-1140-4393-a689-af8264e97b43" (UID: "3ce36f61-1140-4393-a689-af8264e97b43"). InnerVolumeSpecName "kube-api-access-9r79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:31.660468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.660441    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util" pod "bbbc60c3-599a-4fde-a5dd-fc899932af6a" (UID: "bbbc60c3-599a-4fde-a5dd-fc899932af6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.661423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.661394    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util" pod "3ce36f61-1140-4393-a689-af8264e97b43" (UID: "3ce36f61-1140-4393-a689-af8264e97b43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.662101 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.662067    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util" pod "cd122dd8-3aec-407b-9407-7039473477f4" (UID: "cd122dd8-3aec-407b-9407-7039473477f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:31.754911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754891    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.754911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754911    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754919    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbbc60c3-599a-4fde-a5dd-fc899932af6a-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754928    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6xc\" (UniqueName: \"kubernetes.io/projected/cd122dd8-3aec-407b-9407-7039473477f4-kube-api-access-9j6xc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754937    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbzs\" (UniqueName: \"kubernetes.io/projected/bbbc60c3-599a-4fde-a5dd-fc899932af6a-kube-api-access-qgbzs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754946    5065 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-util\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754954    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r79q\" (UniqueName: \"kubernetes.io/projected/3ce36f61-1140-4393-a689-af8264e97b43-kube-api-access-9r79q\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754962    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd122dd8-3aec-407b-9407-7039473477f4-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:31.755033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:31.754973    5065 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ce36f61-1140-4393-a689-af8264e97b43-bundle\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:32.341866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.341841    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5"
Oct 30 18:37:32.342264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.341833    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5bmb5" event={"ID":"3ce36f61-1140-4393-a689-af8264e97b43","Type":"ContainerDied","Data":"710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860"}
Oct 30 18:37:32.342264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.341981    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710a13900c07fa9f456afa018b97b27783838c435ad96f4c75c75cfba7132860"
Oct 30 18:37:32.344758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.344726    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4" event={"ID":"bbbc60c3-599a-4fde-a5dd-fc899932af6a","Type":"ContainerDied","Data":"6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb"}
Oct 30 18:37:32.344887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.344764    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6b9t4"
Oct 30 18:37:32.344887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.344774    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c9110f938f2babb74994f00f58bfb57721fb2986b022f42e2b5e89aa01acfdb"
Oct 30 18:37:32.347266 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.347232    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt" event={"ID":"cd122dd8-3aec-407b-9407-7039473477f4","Type":"ContainerDied","Data":"1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4"}
Oct 30 18:37:32.347266 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.347268    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bde5ac0a25349bfcfbf7e209ca8f2f947a5e7c9a39d9885acf3b6a87519e3d4"
Oct 30 18:37:32.347403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:32.347389    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dwlt"
Oct 30 18:37:41.488030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488001    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"]
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488325    5065 cpu_manager.go:414] "containerMap: removing container" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488336    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488345    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488372    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488386    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488391    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488398    5065 cpu_manager.go:414] "containerMap: removing container" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488402    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488411    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488416    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488422    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488427    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488433    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488438    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488444    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488449    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488455    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488460    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488465    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488469    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488477    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488484    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="util"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488489    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488494    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488504    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488508    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488513    5065 cpu_manager.go:414] "containerMap: removing container" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488517    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488527    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488531    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488539    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488543    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="pull"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488549    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488553    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488558    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488562    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488605    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce36f61-1140-4393-a689-af8264e97b43" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488615    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a5bc43-7467-4d2e-b234-b7460228b455" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488622    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b91b10-947a-476b-a5a0-cd1cedf8ebc3" containerName="extract"
Oct 30 18:37:41.488606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488627    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc60c3-599a-4fde-a5dd-fc899932af6a" containerName="extract"
Oct 30 18:37:41.489743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488633    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd122dd8-3aec-407b-9407-7039473477f4" containerName="extract"
Oct 30 18:37:41.489743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.488639    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0379437a-e747-4eb0-b1e0-176d617c36c2" containerName="extract"
Oct 30 18:37:41.491589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.491562    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:41.508982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.508966    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 18:37:41.509133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.509076    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:37:41.509208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.509197    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:37:41.509468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.509455    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rtwlm\""
Oct 30 18:37:41.521239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.521213    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"]
Oct 30 18:37:41.521665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.521643    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkh29\" (UniqueName: \"kubernetes.io/projected/069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4-kube-api-access-jkh29\") pod \"dns-operator-controller-manager-8557cbdb84-dpdm4\" (UID: \"069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:41.524746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.524727    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"]
Oct 30 18:37:41.524850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.524810    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.590062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.590036    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 18:37:41.590248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.590230    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 18:37:41.606771 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.606749    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:37:41.606856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.606823    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:37:41.607122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.607108    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:37:41.607122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.607110    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c242x\""
Oct 30 18:37:41.622533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622511    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc26z\" (UniqueName: \"kubernetes.io/projected/0af971e4-a728-482f-b773-fb48d53f049e-kube-api-access-tc26z\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.622581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622538    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.622581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622565    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkh29\" (UniqueName: \"kubernetes.io/projected/069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4-kube-api-access-jkh29\") pod \"dns-operator-controller-manager-8557cbdb84-dpdm4\" (UID: \"069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:41.622659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622626    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.622703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622662    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.622789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622758    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.622836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.622822    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.643435 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.643411    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"]
Oct 30 18:37:41.674421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.674399    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkh29\" (UniqueName: \"kubernetes.io/projected/069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4-kube-api-access-jkh29\") pod \"dns-operator-controller-manager-8557cbdb84-dpdm4\" (UID: \"069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:41.723114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723072    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723125    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723148    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723183    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723207    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc26z\" (UniqueName: \"kubernetes.io/projected/0af971e4-a728-482f-b773-fb48d53f049e-kube-api-access-tc26z\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723223    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.723620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.723592    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.725553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.725530    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.725657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.725530    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.725657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.725538    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0af971e4-a728-482f-b773-fb48d53f049e-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.725657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.725629    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af971e4-a728-482f-b773-fb48d53f049e-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.768985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.768966    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc26z\" (UniqueName: \"kubernetes.io/projected/0af971e4-a728-482f-b773-fb48d53f049e-kube-api-access-tc26z\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-s56s5\" (UID: \"0af971e4-a728-482f-b773-fb48d53f049e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:41.803767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.803748    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:41.838988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:41.838965    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:42.134867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:42.134809    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"]
Oct 30 18:37:42.137307 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:42.137282    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069f3fdc_c1ae_47d3_ad80_9b1b169bc3f4.slice/crio-738e7d0e5c140cccd680c185da0e9962b866b66e54b2fa632e7615f461414057 WatchSource:0}: Error finding container 738e7d0e5c140cccd680c185da0e9962b866b66e54b2fa632e7615f461414057: Status 404 returned error can't find the container with id 738e7d0e5c140cccd680c185da0e9962b866b66e54b2fa632e7615f461414057
Oct 30 18:37:42.236642 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:42.236616    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"]
Oct 30 18:37:42.239605 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:42.239574    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af971e4_a728_482f_b773_fb48d53f049e.slice/crio-84babaa4131ec7759f4ced6f46f1bf589b2f7739c95db59041cf064c22cf8b77 WatchSource:0}: Error finding container 84babaa4131ec7759f4ced6f46f1bf589b2f7739c95db59041cf064c22cf8b77: Status 404 returned error can't find the container with id 84babaa4131ec7759f4ced6f46f1bf589b2f7739c95db59041cf064c22cf8b77
Oct 30 18:37:42.394327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:42.394258    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4" event={"ID":"069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4","Type":"ContainerStarted","Data":"738e7d0e5c140cccd680c185da0e9962b866b66e54b2fa632e7615f461414057"}
Oct 30 18:37:42.395772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:42.395749    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5" event={"ID":"0af971e4-a728-482f-b773-fb48d53f049e","Type":"ContainerStarted","Data":"84babaa4131ec7759f4ced6f46f1bf589b2f7739c95db59041cf064c22cf8b77"}
Oct 30 18:37:45.053538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.053502    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"]
Oct 30 18:37:45.057856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.057833    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:45.072874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.072848    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f4pg2\""
Oct 30 18:37:45.100050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.099987    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"]
Oct 30 18:37:45.151174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.151129    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt8g\" (UniqueName: \"kubernetes.io/projected/29e00d0c-30b8-45fd-9b1a-a26e6bed0594-kube-api-access-4qt8g\") pod \"limitador-operator-controller-manager-5778bb4449-f8r52\" (UID: \"29e00d0c-30b8-45fd-9b1a-a26e6bed0594\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:45.252431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.252396    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt8g\" (UniqueName: \"kubernetes.io/projected/29e00d0c-30b8-45fd-9b1a-a26e6bed0594-kube-api-access-4qt8g\") pod \"limitador-operator-controller-manager-5778bb4449-f8r52\" (UID: \"29e00d0c-30b8-45fd-9b1a-a26e6bed0594\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:45.271567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.271536    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt8g\" (UniqueName: \"kubernetes.io/projected/29e00d0c-30b8-45fd-9b1a-a26e6bed0594-kube-api-access-4qt8g\") pod \"limitador-operator-controller-manager-5778bb4449-f8r52\" (UID: \"29e00d0c-30b8-45fd-9b1a-a26e6bed0594\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:45.374696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.374626    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:45.698698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:45.698677    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"]
Oct 30 18:37:45.700127 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:45.700077    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e00d0c_30b8_45fd_9b1a_a26e6bed0594.slice/crio-51f5919f95a7942c24549256b3a1167b1a99c041ac8070cfe3940790c226c89d WatchSource:0}: Error finding container 51f5919f95a7942c24549256b3a1167b1a99c041ac8070cfe3940790c226c89d: Status 404 returned error can't find the container with id 51f5919f95a7942c24549256b3a1167b1a99c041ac8070cfe3940790c226c89d
Oct 30 18:37:46.424163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.424124    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4" event={"ID":"069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4","Type":"ContainerStarted","Data":"d58315a163c7ea45fb41e6332f5c12659716f97b52796d7b0d8f08672e50e47a"}
Oct 30 18:37:46.424554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.424248    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:46.425962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.425937    5065 generic.go:356] "Generic (PLEG): container finished" podID="0af971e4-a728-482f-b773-fb48d53f049e" containerID="788879cf7272acc116dfcb5f18e072f18acdfd831a321a79ce6c3ed38becb449" exitCode=0
Oct 30 18:37:46.426080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.425971    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5" event={"ID":"0af971e4-a728-482f-b773-fb48d53f049e","Type":"ContainerDied","Data":"788879cf7272acc116dfcb5f18e072f18acdfd831a321a79ce6c3ed38becb449"}
Oct 30 18:37:46.427636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.427612    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52" event={"ID":"29e00d0c-30b8-45fd-9b1a-a26e6bed0594","Type":"ContainerStarted","Data":"51f5919f95a7942c24549256b3a1167b1a99c041ac8070cfe3940790c226c89d"}
Oct 30 18:37:46.492151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:46.491042    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4" podStartSLOduration=2.236221746 podStartE2EDuration="5.491027933s" totalImagesPullingTime="3.254806187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:46.490114838 +0000 UTC m=+286.753579810" watchObservedRunningTime="2026-10-30 18:37:46.491027933 +0000 UTC m=+286.754492880"
Oct 30 18:37:47.434650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.434620    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5" event={"ID":"0af971e4-a728-482f-b773-fb48d53f049e","Type":"ContainerStarted","Data":"2b8d0e674c8ebc938a0a293fda3b55b9cb71af911990cd5e82d0e0b8a2fd484d"}
Oct 30 18:37:47.435151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.434720    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:47.436736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.436704    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52" event={"ID":"29e00d0c-30b8-45fd-9b1a-a26e6bed0594","Type":"ContainerStarted","Data":"38f1b77c92f86b0bce866d54a99b9ee1186c483fcb2e569db8dd4d5a5da1ba5a"}
Oct 30 18:37:47.436849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.436772    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:47.655659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.655552    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5" podStartSLOduration=2.363248011 podStartE2EDuration="6.65553313s" totalImagesPullingTime="3.823448689s" totalInitContainerRuntime="468.83643ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.650808304 +0000 UTC m=+287.914273279" watchObservedRunningTime="2026-10-30 18:37:47.65553313 +0000 UTC m=+287.918998075"
Oct 30 18:37:47.730899 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:47.730842    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52" podStartSLOduration=1.158354466 podStartE2EDuration="2.730830396s" totalImagesPullingTime="1.57247593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.730275552 +0000 UTC m=+287.993740493" watchObservedRunningTime="2026-10-30 18:37:47.730830396 +0000 UTC m=+287.994295343"
Oct 30 18:37:53.058186 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:37:53.058139    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:37:54.674665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.674632    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"]
Oct 30 18:37:54.683051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.683019    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.703249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.703228    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v2bss\""
Oct 30 18:37:54.727347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.727323    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.727572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.727547    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqqv\" (UniqueName: \"kubernetes.io/projected/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-kube-api-access-5qqqv\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.816566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.816543    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"]
Oct 30 18:37:54.828010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.827985    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.828057    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqqv\" (UniqueName: \"kubernetes.io/projected/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-kube-api-access-5qqqv\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.828355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.828336    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.857028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.857008    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqqv\" (UniqueName: \"kubernetes.io/projected/af0fa85d-ed63-4c01-9a04-0ee25ca545a1-kube-api-access-5qqqv\") pod \"kuadrant-operator-controller-manager-678d87766f-dwsqg\" (UID: \"af0fa85d-ed63-4c01-9a04-0ee25ca545a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:54.995107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:54.995037    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:37:55.309227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:55.309203    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"]
Oct 30 18:37:55.311639 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:37:55.311604    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0fa85d_ed63_4c01_9a04_0ee25ca545a1.slice/crio-1d1597fa828eaead691868b75beae4302fde14bd94a86ed3dd1c3b23b3c6776a WatchSource:0}: Error finding container 1d1597fa828eaead691868b75beae4302fde14bd94a86ed3dd1c3b23b3c6776a: Status 404 returned error can't find the container with id 1d1597fa828eaead691868b75beae4302fde14bd94a86ed3dd1c3b23b3c6776a
Oct 30 18:37:55.474796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:55.474764    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg" event={"ID":"af0fa85d-ed63-4c01-9a04-0ee25ca545a1","Type":"ContainerStarted","Data":"1d1597fa828eaead691868b75beae4302fde14bd94a86ed3dd1c3b23b3c6776a"}
Oct 30 18:37:57.439406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:57.439378    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dpdm4"
Oct 30 18:37:58.444248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:58.444218    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-s56s5"
Oct 30 18:37:58.444696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:58.444298    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f8r52"
Oct 30 18:37:59.916729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:59.916705    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:37:59.917125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:59.917014    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:37:59.923944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:59.923926    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:37:59.924300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:37:59.924286    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:38:00.502828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:00.502792    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg" event={"ID":"af0fa85d-ed63-4c01-9a04-0ee25ca545a1","Type":"ContainerStarted","Data":"6da456f3f254ae6aa50e6a3b50b5b0455cb0b6dc564ac0745f5802faf2dec984"}
Oct 30 18:38:00.503022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:00.502865    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:38:00.571796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:00.571745    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg" podStartSLOduration=2.428469734 podStartE2EDuration="6.571731979s" totalImagesPullingTime="4.143262245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:00.567976951 +0000 UTC m=+300.831441878" watchObservedRunningTime="2026-10-30 18:38:00.571731979 +0000 UTC m=+300.835196928"
Oct 30 18:38:11.510526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:11.510444    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dwsqg"
Oct 30 18:38:12.818764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.818729    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"]
Oct 30 18:38:12.822958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.822935    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.853342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.853312    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"]
Oct 30 18:38:12.859119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.853742    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a3778ff-4333-47d7-a887-a8a669a192d8-tmp\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.859119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.853796    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.859119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.853827    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv57\" (UniqueName: \"kubernetes.io/projected/9a3778ff-4333-47d7-a887-a8a669a192d8-kube-api-access-4wv57\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.869985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.869958    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 18:38:12.878186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.878163    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 18:38:12.878297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.878220    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9s2fw\""
Oct 30 18:38:12.954381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.954359    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.954473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.954397    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv57\" (UniqueName: \"kubernetes.io/projected/9a3778ff-4333-47d7-a887-a8a669a192d8-kube-api-access-4wv57\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.954473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.954441    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a3778ff-4333-47d7-a887-a8a669a192d8-tmp\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:12.954549 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:38:12.954495    5065 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Oct 30 18:38:12.954579 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:38:12.954570    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs podName:9a3778ff-4333-47d7-a887-a8a669a192d8 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:13.454554729 +0000 UTC m=+313.718019678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs") pod "kube-auth-proxy-65c8c79996-7k7ts" (UID: "9a3778ff-4333-47d7-a887-a8a669a192d8") : secret "kube-auth-proxy-tls" not found
Oct 30 18:38:12.956885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:12.956863    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a3778ff-4333-47d7-a887-a8a669a192d8-tmp\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:13.026436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:13.026402    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv57\" (UniqueName: \"kubernetes.io/projected/9a3778ff-4333-47d7-a887-a8a669a192d8-kube-api-access-4wv57\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:13.458530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:13.458495    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:13.461067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:13.461043    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3778ff-4333-47d7-a887-a8a669a192d8-tls-certs\") pod \"kube-auth-proxy-65c8c79996-7k7ts\" (UID: \"9a3778ff-4333-47d7-a887-a8a669a192d8\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:13.739230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:13.739152    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"
Oct 30 18:38:14.054744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:14.054622    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts"]
Oct 30 18:38:14.057421 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:38:14.057395    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3778ff_4333_47d7_a887_a8a669a192d8.slice/crio-d89cbbfd4b175b24f59066c9fdf30e48f5c67676fd452489e4012361a643c314 WatchSource:0}: Error finding container d89cbbfd4b175b24f59066c9fdf30e48f5c67676fd452489e4012361a643c314: Status 404 returned error can't find the container with id d89cbbfd4b175b24f59066c9fdf30e48f5c67676fd452489e4012361a643c314
Oct 30 18:38:14.059543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:14.059518    5065 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:38:14.579950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:14.579911    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts" event={"ID":"9a3778ff-4333-47d7-a887-a8a669a192d8","Type":"ContainerStarted","Data":"d89cbbfd4b175b24f59066c9fdf30e48f5c67676fd452489e4012361a643c314"}
Oct 30 18:38:17.595188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:17.595150    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts" event={"ID":"9a3778ff-4333-47d7-a887-a8a669a192d8","Type":"ContainerStarted","Data":"daa0e1582947798d6905d837529882761766d621deb1bdd34334581a7b852c0a"}
Oct 30 18:38:17.631330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:17.631284    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65c8c79996-7k7ts" podStartSLOduration=3.02297604 podStartE2EDuration="5.631270254s" totalImagesPullingTime="2.608294214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:17.629109536 +0000 UTC m=+317.892574485" watchObservedRunningTime="2026-10-30 18:38:17.631270254 +0000 UTC m=+317.894735202"
Oct 30 18:38:18.701708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.701674    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"]
Oct 30 18:38:18.746833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.746808    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.761836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.761817    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-rwvrc\""
Oct 30 18:38:18.764895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.764871    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"]
Oct 30 18:38:18.781247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.781230    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:38:18.781589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.781570    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:38:18.786568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.786547    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 18:38:18.804876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.804858    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.804976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.804892    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmgz\" (UniqueName: \"kubernetes.io/projected/33068276-152a-4e10-bce0-8b177f00d071-kube-api-access-sjmgz\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.804976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.804925    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.805071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.805056    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33068276-152a-4e10-bce0-8b177f00d071-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905506    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33068276-152a-4e10-bce0-8b177f00d071-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905543    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905566    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmgz\" (UniqueName: \"kubernetes.io/projected/33068276-152a-4e10-bce0-8b177f00d071-kube-api-access-sjmgz\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905596    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905915    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.905969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.905945    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33068276-152a-4e10-bce0-8b177f00d071-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.906419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.906396    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33068276-152a-4e10-bce0-8b177f00d071-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:18.937733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:18.937709    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmgz\" (UniqueName: \"kubernetes.io/projected/33068276-152a-4e10-bce0-8b177f00d071-kube-api-access-sjmgz\") pod \"kserve-module-controller-manager-7566cf5fc8-vnk4q\" (UID: \"33068276-152a-4e10-bce0-8b177f00d071\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:19.069182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:19.069158    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:19.368824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:19.368801    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"]
Oct 30 18:38:19.370955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:38:19.370926    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33068276_152a_4e10_bce0_8b177f00d071.slice/crio-934e83c8956fdc0aa6ee79f85af77de1c36564c9f33aded1655a26757f4b3bd3 WatchSource:0}: Error finding container 934e83c8956fdc0aa6ee79f85af77de1c36564c9f33aded1655a26757f4b3bd3: Status 404 returned error can't find the container with id 934e83c8956fdc0aa6ee79f85af77de1c36564c9f33aded1655a26757f4b3bd3
Oct 30 18:38:19.605758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:19.605730    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q" event={"ID":"33068276-152a-4e10-bce0-8b177f00d071","Type":"ContainerStarted","Data":"934e83c8956fdc0aa6ee79f85af77de1c36564c9f33aded1655a26757f4b3bd3"}
Oct 30 18:38:21.616813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:21.616778    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q" event={"ID":"33068276-152a-4e10-bce0-8b177f00d071","Type":"ContainerStarted","Data":"4705855e3e07994a52fc3265a57452285b08c5a70055f4387a2992da793cc674"}
Oct 30 18:38:21.617274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:21.616934    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:21.702540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:21.702500    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q" podStartSLOduration=2.293903432 podStartE2EDuration="3.702486783s" totalImagesPullingTime="1.408583351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:21.701291351 +0000 UTC m=+321.964756370" watchObservedRunningTime="2026-10-30 18:38:21.702486783 +0000 UTC m=+321.965951734"
Oct 30 18:38:23.057441 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:38:23.057404    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:38:26.590069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.590036    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:26.593876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.593858    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:26.607604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.607582    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-255rx\""
Oct 30 18:38:26.614214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.614192    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:26.666037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.666011    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mrg\" (UniqueName: \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\") pod \"authorino-57f7f8659f-4m2m8\" (UID: \"46c5f005-3137-4c1b-ba41-1fe4a42924c2\") " pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:26.766735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.766710    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mrg\" (UniqueName: \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\") pod \"authorino-57f7f8659f-4m2m8\" (UID: \"46c5f005-3137-4c1b-ba41-1fe4a42924c2\") " pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:26.795819 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.795802    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mrg\" (UniqueName: \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\") pod \"authorino-57f7f8659f-4m2m8\" (UID: \"46c5f005-3137-4c1b-ba41-1fe4a42924c2\") " pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:26.927970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:26.927925    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:27.206931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:27.206872    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:27.210276 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:38:27.210245    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46c5f005_3137_4c1b_ba41_1fe4a42924c2.slice/crio-6d9003d11323a73469fde79626fe64281e45acbebdf762ee6019f6ed62ae9a41 WatchSource:0}: Error finding container 6d9003d11323a73469fde79626fe64281e45acbebdf762ee6019f6ed62ae9a41: Status 404 returned error can't find the container with id 6d9003d11323a73469fde79626fe64281e45acbebdf762ee6019f6ed62ae9a41
Oct 30 18:38:27.649533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:27.649503    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" event={"ID":"46c5f005-3137-4c1b-ba41-1fe4a42924c2","Type":"ContainerStarted","Data":"6d9003d11323a73469fde79626fe64281e45acbebdf762ee6019f6ed62ae9a41"}
Oct 30 18:38:29.662001 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:29.661960    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" event={"ID":"46c5f005-3137-4c1b-ba41-1fe4a42924c2","Type":"ContainerStarted","Data":"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"}
Oct 30 18:38:29.687154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:29.687114    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" podStartSLOduration=1.9918258880000002 podStartE2EDuration="3.687079695s" totalImagesPullingTime="1.695253807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:29.686435046 +0000 UTC m=+329.949899996" watchObservedRunningTime="2026-10-30 18:38:29.687079695 +0000 UTC m=+329.950544643"
Oct 30 18:38:29.868826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:29.868784    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:31.671824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:31.671780    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" podUID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" containerName="authorino" containerID="cri-o://e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f" gracePeriod=30
Oct 30 18:38:31.914852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:31.914833    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:32.007745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.007676    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\" (UniqueName: \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\") pod \"46c5f005-3137-4c1b-ba41-1fe4a42924c2\" (UID: \"46c5f005-3137-4c1b-ba41-1fe4a42924c2\") "
Oct 30 18:38:32.009893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.009869    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg" pod "46c5f005-3137-4c1b-ba41-1fe4a42924c2" (UID: "46c5f005-3137-4c1b-ba41-1fe4a42924c2"). InnerVolumeSpecName "kube-api-access-g8mrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:32.108794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.108766    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8mrg\" (UniqueName: \"kubernetes.io/projected/46c5f005-3137-4c1b-ba41-1fe4a42924c2-kube-api-access-g8mrg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:32.678587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.678550    5065 generic.go:356] "Generic (PLEG): container finished" podID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" containerID="e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f" exitCode=0
Oct 30 18:38:32.678976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.678608    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4m2m8"
Oct 30 18:38:32.678976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.678642    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" event={"ID":"46c5f005-3137-4c1b-ba41-1fe4a42924c2","Type":"ContainerDied","Data":"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"}
Oct 30 18:38:32.678976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.678697    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4m2m8" event={"ID":"46c5f005-3137-4c1b-ba41-1fe4a42924c2","Type":"ContainerDied","Data":"6d9003d11323a73469fde79626fe64281e45acbebdf762ee6019f6ed62ae9a41"}
Oct 30 18:38:32.678976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.678718    5065 scope.go:122] "RemoveContainer" containerID="e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"
Oct 30 18:38:32.688575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.688558    5065 scope.go:122] "RemoveContainer" containerID="e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"
Oct 30 18:38:32.688903 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:38:32.688880    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f\": container with ID starting with e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f not found: ID does not exist" containerID="e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"
Oct 30 18:38:32.688956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.688911    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f"} err="failed to get container status \"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f\": rpc error: code = NotFound desc = could not find container \"e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f\": container with ID starting with e15d50fb0be45aef65ebd3caf22a01729b740141f6916e7fe5188924d197f33f not found: ID does not exist"
Oct 30 18:38:32.703376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.703355    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:32.705457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:32.705439    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4m2m8"]
Oct 30 18:38:33.887412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:33.887362    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" path="/var/lib/kubelet/pods/46c5f005-3137-4c1b-ba41-1fe4a42924c2/volumes"
Oct 30 18:38:36.536365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.536336    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"]
Oct 30 18:38:36.536870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.536853    5065 cpu_manager.go:414] "containerMap: removing container" podUID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" containerName="authorino"
Oct 30 18:38:36.536912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.536873    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" containerName="authorino"
Oct 30 18:38:36.536990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.536979    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c5f005-3137-4c1b-ba41-1fe4a42924c2" containerName="authorino"
Oct 30 18:38:36.541029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.541009    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.556501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.556476    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:38:36.556791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.556769    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:38:36.557016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.556983    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 18:38:36.557274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.557257    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-m9tz5\""
Oct 30 18:38:36.558648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.558627    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"]
Oct 30 18:38:36.649035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.649010    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsv9\" (UniqueName: \"kubernetes.io/projected/46c4e3cc-8401-41d5-83f8-7ea5632ef388-kube-api-access-znsv9\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.649159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.649046    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46c4e3cc-8401-41d5-83f8-7ea5632ef388-data\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.749353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.749329    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsv9\" (UniqueName: \"kubernetes.io/projected/46c4e3cc-8401-41d5-83f8-7ea5632ef388-kube-api-access-znsv9\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.749439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.749364    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46c4e3cc-8401-41d5-83f8-7ea5632ef388-data\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.749736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.749720    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46c4e3cc-8401-41d5-83f8-7ea5632ef388-data\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.763298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.763270    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsv9\" (UniqueName: \"kubernetes.io/projected/46c4e3cc-8401-41d5-83f8-7ea5632ef388-kube-api-access-znsv9\") pod \"postgres-7cdbf6dc44-t4gbc\" (UID: \"46c4e3cc-8401-41d5-83f8-7ea5632ef388\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:36.854043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:36.853994    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:37.132747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:37.132717    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"]
Oct 30 18:38:37.135037 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:38:37.134999    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c4e3cc_8401_41d5_83f8_7ea5632ef388.slice/crio-12ee1ec49faf4a08b654bb9a536750af701b2d60896c292288fadb69c00e631c WatchSource:0}: Error finding container 12ee1ec49faf4a08b654bb9a536750af701b2d60896c292288fadb69c00e631c: Status 404 returned error can't find the container with id 12ee1ec49faf4a08b654bb9a536750af701b2d60896c292288fadb69c00e631c
Oct 30 18:38:37.705994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:37.705960    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc" event={"ID":"46c4e3cc-8401-41d5-83f8-7ea5632ef388","Type":"ContainerStarted","Data":"12ee1ec49faf4a08b654bb9a536750af701b2d60896c292288fadb69c00e631c"}
Oct 30 18:38:42.623039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:42.623019    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vnk4q"
Oct 30 18:38:42.737061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:42.737027    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc" event={"ID":"46c4e3cc-8401-41d5-83f8-7ea5632ef388","Type":"ContainerStarted","Data":"bbda16532aa7f52061016aa576b789b84597104938bb52762ec39753550ef51e"}
Oct 30 18:38:42.737256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:42.737120    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:42.758522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:42.758473    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc" podStartSLOduration=1.320827942 podStartE2EDuration="6.758459061s" totalImagesPullingTime="5.437631119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:42.75684628 +0000 UTC m=+343.020311230" watchObservedRunningTime="2026-10-30 18:38:42.758459061 +0000 UTC m=+343.021924025"
Oct 30 18:38:48.776030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:38:48.776003    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc"
Oct 30 18:38:53.055630 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:38:53.055591    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:39:11.431217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.431184    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6g8zb"]
Oct 30 18:39:11.434970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.434948    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.447737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.447719    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 18:39:11.448971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.448956    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rtt8n\""
Oct 30 18:39:11.456964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.456937    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6g8zb"]
Oct 30 18:39:11.511389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.511359    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2bnm"]
Oct 30 18:39:11.515431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.515411    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.520844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.520821    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-metrics-certs\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.521131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.521060    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-cert\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.521250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.521182    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7408db0f-b1e1-4ef0-ab31-43955955d798-cert\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.521312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.521242    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjff\" (UniqueName: \"kubernetes.io/projected/6a46c63a-9f3b-4668-af47-a9cc278533e9-kube-api-access-8fjff\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.521312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.521294    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmmc\" (UniqueName: \"kubernetes.io/projected/7408db0f-b1e1-4ef0-ab31-43955955d798-kube-api-access-qhmmc\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.528027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.528008    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2bnm"]
Oct 30 18:39:11.546835 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.546767    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:39:11.546985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.546942    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:39:11.547171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.547152    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-m62qh\""
Oct 30 18:39:11.622681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.622652    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-metrics-certs\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.622821 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.622688    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-cert\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.622821 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.622707    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7408db0f-b1e1-4ef0-ab31-43955955d798-cert\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.622821 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.622725    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjff\" (UniqueName: \"kubernetes.io/projected/6a46c63a-9f3b-4668-af47-a9cc278533e9-kube-api-access-8fjff\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.622984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.622858    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmmc\" (UniqueName: \"kubernetes.io/projected/7408db0f-b1e1-4ef0-ab31-43955955d798-kube-api-access-qhmmc\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.625415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.625390    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-cert\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.625510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.625426    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a46c63a-9f3b-4668-af47-a9cc278533e9-metrics-certs\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.625510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.625498    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7408db0f-b1e1-4ef0-ab31-43955955d798-cert\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.635679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.635658    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmmc\" (UniqueName: \"kubernetes.io/projected/7408db0f-b1e1-4ef0-ab31-43955955d798-kube-api-access-qhmmc\") pod \"kserve-controller-manager-776cc66d47-6g8zb\" (UID: \"7408db0f-b1e1-4ef0-ab31-43955955d798\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.635854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.635837    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjff\" (UniqueName: \"kubernetes.io/projected/6a46c63a-9f3b-4668-af47-a9cc278533e9-kube-api-access-8fjff\") pod \"odh-model-controller-769875db56-z2bnm\" (UID: \"6a46c63a-9f3b-4668-af47-a9cc278533e9\") " pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:11.750235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.750184    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:11.829823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:11.829796    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:12.040098 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:12.040065    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6g8zb"]
Oct 30 18:39:12.041912 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:39:12.041887    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7408db0f_b1e1_4ef0_ab31_43955955d798.slice/crio-14371ba5556a8a01cf3176ef0f7716692b29889f720ae90ff012858fb21df1e1 WatchSource:0}: Error finding container 14371ba5556a8a01cf3176ef0f7716692b29889f720ae90ff012858fb21df1e1: Status 404 returned error can't find the container with id 14371ba5556a8a01cf3176ef0f7716692b29889f720ae90ff012858fb21df1e1
Oct 30 18:39:12.132189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:12.132166    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2bnm"]
Oct 30 18:39:12.134509 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:39:12.134477    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a46c63a_9f3b_4668_af47_a9cc278533e9.slice/crio-6ba9e9c2004f0ff2f98d18d61dc900f5cf75d837cd43e4cd58704f8867128ecc WatchSource:0}: Error finding container 6ba9e9c2004f0ff2f98d18d61dc900f5cf75d837cd43e4cd58704f8867128ecc: Status 404 returned error can't find the container with id 6ba9e9c2004f0ff2f98d18d61dc900f5cf75d837cd43e4cd58704f8867128ecc
Oct 30 18:39:12.902932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:12.902816    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z2bnm" event={"ID":"6a46c63a-9f3b-4668-af47-a9cc278533e9","Type":"ContainerStarted","Data":"6ba9e9c2004f0ff2f98d18d61dc900f5cf75d837cd43e4cd58704f8867128ecc"}
Oct 30 18:39:12.907041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:12.907012    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb" event={"ID":"7408db0f-b1e1-4ef0-ab31-43955955d798","Type":"ContainerStarted","Data":"14371ba5556a8a01cf3176ef0f7716692b29889f720ae90ff012858fb21df1e1"}
Oct 30 18:39:15.924080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.924049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z2bnm" event={"ID":"6a46c63a-9f3b-4668-af47-a9cc278533e9","Type":"ContainerStarted","Data":"ef2e42e7442137df60f610c9180360ede71ded1425dbd22f9c5505e6ddbe5d13"}
Oct 30 18:39:15.924538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.924153    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:15.925694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.925664    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb" event={"ID":"7408db0f-b1e1-4ef0-ab31-43955955d798","Type":"ContainerStarted","Data":"dbf2acc97f1b21dfacf1ffd7aa23a7281d421b894ee56ca7dabb9c43ff3c8e8b"}
Oct 30 18:39:15.925816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.925800    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:15.947814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.947767    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-z2bnm" podStartSLOduration=1.617100716 podStartE2EDuration="4.947755232s" totalImagesPullingTime="3.330654516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.946738955 +0000 UTC m=+376.210203922" watchObservedRunningTime="2026-10-30 18:39:15.947755232 +0000 UTC m=+376.211220224"
Oct 30 18:39:15.974254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:15.974216    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb" podStartSLOduration=2.107617878 podStartE2EDuration="4.974204891s" totalImagesPullingTime="2.866587013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.972949896 +0000 UTC m=+376.236414844" watchObservedRunningTime="2026-10-30 18:39:15.974204891 +0000 UTC m=+376.237669839"
Oct 30 18:39:23.055491 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:39:23.055454    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:39:26.933573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:26.933533    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-z2bnm"
Oct 30 18:39:44.069489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.069457    5065 generic.go:356] "Generic (PLEG): container finished" podID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerID="4d01a3153dd6e2f2e2e64c02dbc6bbbaaca0a1c770e970780841dd279bf93494" exitCode=0
Oct 30 18:39:44.069968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.069540    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerDied","Data":"4d01a3153dd6e2f2e2e64c02dbc6bbbaaca0a1c770e970780841dd279bf93494"}
Oct 30 18:39:44.069968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.069866    5065 scope.go:122] "RemoveContainer" containerID="4d01a3153dd6e2f2e2e64c02dbc6bbbaaca0a1c770e970780841dd279bf93494"
Oct 30 18:39:44.070969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.070939    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:39:44.071036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.071021    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:39:44.071075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.071057    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:39:44.158957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.158932    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:39:45.001110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:44.999481    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-plx6h"]
Oct 30 18:39:45.010805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.010773    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-plx6h"]
Oct 30 18:39:45.011056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.010884    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.015790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.015767    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-255rx\""
Oct 30 18:39:45.069114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.069077    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-plx6h"]
Oct 30 18:39:45.069290 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:39:45.069272    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mwr6z], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-plx6h" podUID="1ac2b156-f51e-4fdd-8a13-e2184ac008f4"
Oct 30 18:39:45.078080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.078051    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerStarted","Data":"e9f506cb7dcce5ed9ab12a9174661c8215dea0a8fec7d46e5ba73c287c07a053"}
Oct 30 18:39:45.078475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.078129    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.112107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.101350    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:45.112743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.112708    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwr6z\" (UniqueName: \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\") pod \"authorino-6f6944bcdd-plx6h\" (UID: \"1ac2b156-f51e-4fdd-8a13-e2184ac008f4\") " pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.114459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.114433    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.116967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.116900    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:45.119078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.119053    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:39:45.119749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.119727    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.213263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.213223    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxhj\" (UniqueName: \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.213457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.213287    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.213697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.213672    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwr6z\" (UniqueName: \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\") pod \"authorino-6f6944bcdd-plx6h\" (UID: \"1ac2b156-f51e-4fdd-8a13-e2184ac008f4\") " pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.227552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.227525    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwr6z\" (UniqueName: \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\") pod \"authorino-6f6944bcdd-plx6h\" (UID: \"1ac2b156-f51e-4fdd-8a13-e2184ac008f4\") " pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:45.314097 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.314061    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\" (UniqueName: \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\") pod \"1ac2b156-f51e-4fdd-8a13-e2184ac008f4\" (UID: \"1ac2b156-f51e-4fdd-8a13-e2184ac008f4\") "
Oct 30 18:39:45.314302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.314269    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxhj\" (UniqueName: \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.314302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.314294    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.316508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.316476    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z" pod "1ac2b156-f51e-4fdd-8a13-e2184ac008f4" (UID: "1ac2b156-f51e-4fdd-8a13-e2184ac008f4"). InnerVolumeSpecName "kube-api-access-mwr6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:45.317118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.317070    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.327987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.327964    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxhj\" (UniqueName: \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\") pod \"authorino-6889cbd8c4-7dkmm\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") " pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.375228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.375203    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:45.375403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.375392    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:45.415231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.415202    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:39:45.415532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.415507    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwr6z\" (UniqueName: \"kubernetes.io/projected/1ac2b156-f51e-4fdd-8a13-e2184ac008f4-kube-api-access-mwr6z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:45.418986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.418966    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.428565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.428540    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:39:45.516272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.516124    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.516272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.516186    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7sd\" (UniqueName: \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.617698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.617615    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.617698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.617678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7sd\" (UniqueName: \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.620499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.620462    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.631714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.631683    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7sd\" (UniqueName: \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\") pod \"authorino-7d7cf978c5-c2scl\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") " pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.677826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.677801    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:45.698109 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:39:45.679301    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09119fce_1a10_4099_a665_e16f0aa782f5.slice/crio-a0399ba9e3c3d0339d98008e30a7c83392b311367bf348a8877b584c20d5d5b5 WatchSource:0}: Error finding container a0399ba9e3c3d0339d98008e30a7c83392b311367bf348a8877b584c20d5d5b5: Status 404 returned error can't find the container with id a0399ba9e3c3d0339d98008e30a7c83392b311367bf348a8877b584c20d5d5b5
Oct 30 18:39:45.736622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.736593    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:39:45.849537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.849509    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:39:45.850325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.850301    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 18:39:45.850439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:45.850354    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 18:39:46.032890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.032871    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:39:46.034652 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:39:46.034622    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ed99ae_c6cc_4da9_b82e_43eb38f0d9cc.slice/crio-e1a482530c09b9776d7ab3a5833d8daea96e94bcf6187b9a1a58f07b33361ca6 WatchSource:0}: Error finding container e1a482530c09b9776d7ab3a5833d8daea96e94bcf6187b9a1a58f07b33361ca6: Status 404 returned error can't find the container with id e1a482530c09b9776d7ab3a5833d8daea96e94bcf6187b9a1a58f07b33361ca6
Oct 30 18:39:46.112533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.112501    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" event={"ID":"09119fce-1a10-4099-a665-e16f0aa782f5","Type":"ContainerStarted","Data":"a0399ba9e3c3d0339d98008e30a7c83392b311367bf348a8877b584c20d5d5b5"}
Oct 30 18:39:46.113824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.113790    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" event={"ID":"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc","Type":"ContainerStarted","Data":"e1a482530c09b9776d7ab3a5833d8daea96e94bcf6187b9a1a58f07b33361ca6"}
Oct 30 18:39:46.113824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.113819    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-plx6h"
Oct 30 18:39:46.149020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.148996    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-plx6h"]
Oct 30 18:39:46.152925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.152888    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-plx6h"]
Oct 30 18:39:46.849597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.849562    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 18:39:46.849777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.849617    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 18:39:46.936797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:46.936725    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6g8zb"
Oct 30 18:39:47.122357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.122322    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" event={"ID":"09119fce-1a10-4099-a665-e16f0aa782f5","Type":"ContainerStarted","Data":"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"}
Oct 30 18:39:47.122357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.122351    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" podUID="09119fce-1a10-4099-a665-e16f0aa782f5" containerName="authorino" containerID="cri-o://50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5" gracePeriod=30
Oct 30 18:39:47.147398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.147347    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" podStartSLOduration=1.223498932 podStartE2EDuration="2.147329211s" totalImagesPullingTime="923.830279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:47.145315071 +0000 UTC m=+407.408780020" watchObservedRunningTime="2026-10-30 18:39:47.147329211 +0000 UTC m=+407.410794158"
Oct 30 18:39:47.490955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.487046    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:47.637352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.637324    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\" (UniqueName: \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\") pod \"09119fce-1a10-4099-a665-e16f0aa782f5\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") "
Oct 30 18:39:47.637548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.637375    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\" (UniqueName: \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\") pod \"09119fce-1a10-4099-a665-e16f0aa782f5\" (UID: \"09119fce-1a10-4099-a665-e16f0aa782f5\") "
Oct 30 18:39:47.639617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.639572    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj" pod "09119fce-1a10-4099-a665-e16f0aa782f5" (UID: "09119fce-1a10-4099-a665-e16f0aa782f5"). InnerVolumeSpecName "kube-api-access-fcxhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:47.648858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.648827    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert" pod "09119fce-1a10-4099-a665-e16f0aa782f5" (UID: "09119fce-1a10-4099-a665-e16f0aa782f5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:47.738846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.738802    5065 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/09119fce-1a10-4099-a665-e16f0aa782f5-tls-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.738846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.738832    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxhj\" (UniqueName: \"kubernetes.io/projected/09119fce-1a10-4099-a665-e16f0aa782f5-kube-api-access-fcxhj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:47.849381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.849345    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 18:39:47.849545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.849406    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 18:39:47.906112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:47.900678    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac2b156-f51e-4fdd-8a13-e2184ac008f4" path="/var/lib/kubelet/pods/1ac2b156-f51e-4fdd-8a13-e2184ac008f4/volumes"
Oct 30 18:39:48.129483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.129396    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" event={"ID":"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc","Type":"ContainerStarted","Data":"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"}
Oct 30 18:39:48.130985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.130953    5065 generic.go:356] "Generic (PLEG): container finished" podID="09119fce-1a10-4099-a665-e16f0aa782f5" containerID="50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5" exitCode=0
Oct 30 18:39:48.131158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.130999    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm"
Oct 30 18:39:48.131158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.131016    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" event={"ID":"09119fce-1a10-4099-a665-e16f0aa782f5","Type":"ContainerDied","Data":"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"}
Oct 30 18:39:48.131158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.131049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7dkmm" event={"ID":"09119fce-1a10-4099-a665-e16f0aa782f5","Type":"ContainerDied","Data":"a0399ba9e3c3d0339d98008e30a7c83392b311367bf348a8877b584c20d5d5b5"}
Oct 30 18:39:48.131158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.131074    5065 scope.go:122] "RemoveContainer" containerID="50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"
Oct 30 18:39:48.140385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.140368    5065 scope.go:122] "RemoveContainer" containerID="50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"
Oct 30 18:39:48.140655 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:39:48.140635    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5\": container with ID starting with 50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5 not found: ID does not exist" containerID="50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"
Oct 30 18:39:48.140726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.140662    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5"} err="failed to get container status \"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5\": rpc error: code = NotFound desc = could not find container \"50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5\": container with ID starting with 50bd1c410211c07d917e6c3823961b1c2d453b27e69a23b841cc14ae050acdd5 not found: ID does not exist"
Oct 30 18:39:48.149171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.149128    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" podStartSLOduration=1.944553799 podStartE2EDuration="3.149114515s" totalImagesPullingTime="1.204560716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:48.148425669 +0000 UTC m=+408.411890617" watchObservedRunningTime="2026-10-30 18:39:48.149114515 +0000 UTC m=+408.412579457"
Oct 30 18:39:48.168156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.168126    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:48.174589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.174569    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7dkmm"]
Oct 30 18:39:48.848783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.848747    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 18:39:48.848957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:48.848804    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 18:39:49.849214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:49.849183    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 18:39:49.849579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:49.849252    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 18:39:49.892654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:49.892618    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09119fce-1a10-4099-a665-e16f0aa782f5" path="/var/lib/kubelet/pods/09119fce-1a10-4099-a665-e16f0aa782f5/volumes"
Oct 30 18:39:50.853788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:50.853756    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:39:50.854276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:50.854109    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:39:50.855029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:39:50.855006    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:39:53.055489 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:39:53.055454    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:40:23.058595 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:40:23.058554    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:40:45.140044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.140016    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"]
Oct 30 18:40:45.140492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.140415    5065 cpu_manager.go:414] "containerMap: removing container" podUID="09119fce-1a10-4099-a665-e16f0aa782f5" containerName="authorino"
Oct 30 18:40:45.140492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.140427    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09119fce-1a10-4099-a665-e16f0aa782f5" containerName="authorino"
Oct 30 18:40:45.140565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.140504    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="09119fce-1a10-4099-a665-e16f0aa782f5" containerName="authorino"
Oct 30 18:40:45.143660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.143640    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:45.155535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.155505    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"]
Oct 30 18:40:45.160358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.160336    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"trustyai-service-operator-controller-manager-dockercfg-qrc5v\""
Oct 30 18:40:45.190154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.190128    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xg7g\" (UniqueName: \"kubernetes.io/projected/b5963b47-ba49-4aa8-a578-f4abe50945d2-kube-api-access-4xg7g\") pod \"trustyai-service-operator-controller-manager-7877dff649-lqdc8\" (UID: \"b5963b47-ba49-4aa8-a578-f4abe50945d2\") " pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:45.291261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.291230    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xg7g\" (UniqueName: \"kubernetes.io/projected/b5963b47-ba49-4aa8-a578-f4abe50945d2-kube-api-access-4xg7g\") pod \"trustyai-service-operator-controller-manager-7877dff649-lqdc8\" (UID: \"b5963b47-ba49-4aa8-a578-f4abe50945d2\") " pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:45.309678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.309651    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xg7g\" (UniqueName: \"kubernetes.io/projected/b5963b47-ba49-4aa8-a578-f4abe50945d2-kube-api-access-4xg7g\") pod \"trustyai-service-operator-controller-manager-7877dff649-lqdc8\" (UID: \"b5963b47-ba49-4aa8-a578-f4abe50945d2\") " pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:45.464847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.464792    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:45.742846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:45.742819    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"]
Oct 30 18:40:45.745051 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:40:45.745020    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5963b47_ba49_4aa8_a578_f4abe50945d2.slice/crio-f80eaf50ca95bbfbfe510464b4f14d75210fe48ddec123dceb89c5072bb80c16 WatchSource:0}: Error finding container f80eaf50ca95bbfbfe510464b4f14d75210fe48ddec123dceb89c5072bb80c16: Status 404 returned error can't find the container with id f80eaf50ca95bbfbfe510464b4f14d75210fe48ddec123dceb89c5072bb80c16
Oct 30 18:40:46.445448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:46.445401    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8" event={"ID":"b5963b47-ba49-4aa8-a578-f4abe50945d2","Type":"ContainerStarted","Data":"f80eaf50ca95bbfbfe510464b4f14d75210fe48ddec123dceb89c5072bb80c16"}
Oct 30 18:40:49.464248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:49.464219    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8" event={"ID":"b5963b47-ba49-4aa8-a578-f4abe50945d2","Type":"ContainerStarted","Data":"be4ccffeedca289ac48c1ad13d5f622f197a2b7472cb5efa3183eb08c018811e"}
Oct 30 18:40:49.464621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:49.464282    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:40:49.486543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:40:49.486496    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8" podStartSLOduration=1.394828984 podStartE2EDuration="4.486482906s" totalImagesPullingTime="3.091653922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:49.485002507 +0000 UTC m=+469.748467482" watchObservedRunningTime="2026-10-30 18:40:49.486482906 +0000 UTC m=+469.749947863"
Oct 30 18:40:53.058900 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:40:53.058856    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:41:00.472330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:00.472292    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/trustyai-service-operator-controller-manager-7877dff649-lqdc8"
Oct 30 18:41:01.916044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.915994    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:01.920121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.920102    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:01.930233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.930213    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 18:41:01.930361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.930257    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 18:41:01.930361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.930310    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hms7p\""
Oct 30 18:41:01.934963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:01.933858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:02.024960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.024932    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzl7\" (UniqueName: \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.025102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.024999    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.125557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.125526    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.125687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.125575    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzl7\" (UniqueName: \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.128178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.128159    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.139621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.139599    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzl7\" (UniqueName: \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\") pod \"ai-gateway-controller-7d85675f98-6kn27\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.234199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.234149    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:02.511449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.511418    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:02.514546 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:02.514517    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d32cc6_fe53_440f_9aa1_053666e47df2.slice/crio-afdb7a5de1b05ef6e4cf98d80e21231b6b69d7046b7565b2156031104062092b WatchSource:0}: Error finding container afdb7a5de1b05ef6e4cf98d80e21231b6b69d7046b7565b2156031104062092b: Status 404 returned error can't find the container with id afdb7a5de1b05ef6e4cf98d80e21231b6b69d7046b7565b2156031104062092b
Oct 30 18:41:02.538384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:02.538357    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" event={"ID":"47d32cc6-fe53-440f-9aa1-053666e47df2","Type":"ContainerStarted","Data":"afdb7a5de1b05ef6e4cf98d80e21231b6b69d7046b7565b2156031104062092b"}
Oct 30 18:41:05.555732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:05.555699    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" event={"ID":"47d32cc6-fe53-440f-9aa1-053666e47df2","Type":"ContainerStarted","Data":"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"}
Oct 30 18:41:05.556111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:05.555761    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:05.580072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:05.580021    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" podStartSLOduration=1.881184212 podStartE2EDuration="4.58000673s" totalImagesPullingTime="2.698822518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:05.57824178 +0000 UTC m=+485.841706727" watchObservedRunningTime="2026-10-30 18:41:05.58000673 +0000 UTC m=+485.843471678"
Oct 30 18:41:06.993676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:06.993643    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wzss6"]
Oct 30 18:41:06.997226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:06.997211    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.010012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.009984    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wzss6"]
Oct 30 18:41:07.011306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.011283    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d9fgz\""
Oct 30 18:41:07.063772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.063744    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015a612c-9ab3-4629-adec-475bca8e56e4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.063886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.063780    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpk5z\" (UniqueName: \"kubernetes.io/projected/015a612c-9ab3-4629-adec-475bca8e56e4-kube-api-access-rpk5z\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.067313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.067295    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 18:41:07.164916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.164879    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015a612c-9ab3-4629-adec-475bca8e56e4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.164916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.164918    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpk5z\" (UniqueName: \"kubernetes.io/projected/015a612c-9ab3-4629-adec-475bca8e56e4-kube-api-access-rpk5z\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.165718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.165699    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/015a612c-9ab3-4629-adec-475bca8e56e4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.178690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.178653    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpk5z\" (UniqueName: \"kubernetes.io/projected/015a612c-9ab3-4629-adec-475bca8e56e4-kube-api-access-rpk5z\") pod \"payload-processing-5f78687d5b-wzss6\" (UID: \"015a612c-9ab3-4629-adec-475bca8e56e4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.311253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.311228    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:07.588622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:07.588598    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wzss6"]
Oct 30 18:41:07.590249 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:07.590223    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015a612c_9ab3_4629_adec_475bca8e56e4.slice/crio-b8e8218b4a3c3cd9d8478fd3cb5bca9ee1d2ddb56ee3862a74f744f63c6becc3 WatchSource:0}: Error finding container b8e8218b4a3c3cd9d8478fd3cb5bca9ee1d2ddb56ee3862a74f744f63c6becc3: Status 404 returned error can't find the container with id b8e8218b4a3c3cd9d8478fd3cb5bca9ee1d2ddb56ee3862a74f744f63c6becc3
Oct 30 18:41:08.573707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:08.573668    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6" event={"ID":"015a612c-9ab3-4629-adec-475bca8e56e4","Type":"ContainerStarted","Data":"b8e8218b4a3c3cd9d8478fd3cb5bca9ee1d2ddb56ee3862a74f744f63c6becc3"}
Oct 30 18:41:10.586607 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:10.586524    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6" event={"ID":"015a612c-9ab3-4629-adec-475bca8e56e4","Type":"ContainerStarted","Data":"7d3952e78e29c24873095681b9147f89580a3b547bb9d9ab73aa4d596ef04725"}
Oct 30 18:41:10.586983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:10.586635    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:10.607823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:10.607778    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6" podStartSLOduration=2.389921397 podStartE2EDuration="4.607767384s" totalImagesPullingTime="2.217845987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:10.606997875 +0000 UTC m=+490.870462824" watchObservedRunningTime="2026-10-30 18:41:10.607767384 +0000 UTC m=+490.871232331"
Oct 30 18:41:16.562915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.562882    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:16.845480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.845401    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"]
Oct 30 18:41:16.849254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.849234    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:16.858751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.858725    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"]
Oct 30 18:41:16.937901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.937879    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f025d9-65d5-4552-bcec-5766fe5f6182-metrics-certs\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:16.938012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:16.937949    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz47z\" (UniqueName: \"kubernetes.io/projected/81f025d9-65d5-4552-bcec-5766fe5f6182-kube-api-access-wz47z\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.039172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.039146    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f025d9-65d5-4552-bcec-5766fe5f6182-metrics-certs\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.039272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.039200    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz47z\" (UniqueName: \"kubernetes.io/projected/81f025d9-65d5-4552-bcec-5766fe5f6182-kube-api-access-wz47z\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.041714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.041689    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f025d9-65d5-4552-bcec-5766fe5f6182-metrics-certs\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.052803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.052779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz47z\" (UniqueName: \"kubernetes.io/projected/81f025d9-65d5-4552-bcec-5766fe5f6182-kube-api-access-wz47z\") pod \"ai-gateway-controller-6f569dfc6b-gn7sj\" (UID: \"81f025d9-65d5-4552-bcec-5766fe5f6182\") " pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.163254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.163199    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:17.438272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.438240    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"]
Oct 30 18:41:17.440278 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:17.440247    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f025d9_65d5_4552_bcec_5766fe5f6182.slice/crio-cf95248cbf6506814978d6bc12712aa4c153a07f694ace7b5674d2fa04a9194c WatchSource:0}: Error finding container cf95248cbf6506814978d6bc12712aa4c153a07f694ace7b5674d2fa04a9194c: Status 404 returned error can't find the container with id cf95248cbf6506814978d6bc12712aa4c153a07f694ace7b5674d2fa04a9194c
Oct 30 18:41:17.623958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:17.623933    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj" event={"ID":"81f025d9-65d5-4552-bcec-5766fe5f6182","Type":"ContainerStarted","Data":"cf95248cbf6506814978d6bc12712aa4c153a07f694ace7b5674d2fa04a9194c"}
Oct 30 18:41:18.631408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:18.631365    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj" event={"ID":"81f025d9-65d5-4552-bcec-5766fe5f6182","Type":"ContainerStarted","Data":"78476702c94292187b99a3af5370797da479759b2325c5a986889cdb4afc8846"}
Oct 30 18:41:18.631831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:18.631477    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:18.656149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:18.656071    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj" podStartSLOduration=2.223611496 podStartE2EDuration="2.656055313s" totalImagesPullingTime="432.443817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:18.65444631 +0000 UTC m=+498.917911271" watchObservedRunningTime="2026-10-30 18:41:18.656055313 +0000 UTC m=+498.919520258"
Oct 30 18:41:21.594357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:21.594320    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-wzss6"
Oct 30 18:41:23.056256 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:41:23.056223    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:41:29.640235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:29.640196    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6f569dfc6b-gn7sj"
Oct 30 18:41:29.691905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:29.691873    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:29.692146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:29.692112    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" podUID="47d32cc6-fe53-440f-9aa1-053666e47df2" containerName="manager" containerID="cri-o://8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337" gracePeriod=30
Oct 30 18:41:29.939264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:29.939235    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:30.036756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.036731    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\") pod \"47d32cc6-fe53-440f-9aa1-053666e47df2\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") "
Oct 30 18:41:30.036907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.036821    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\" (UniqueName: \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\") pod \"47d32cc6-fe53-440f-9aa1-053666e47df2\" (UID: \"47d32cc6-fe53-440f-9aa1-053666e47df2\") "
Oct 30 18:41:30.039076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.039049    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs" pod "47d32cc6-fe53-440f-9aa1-053666e47df2" (UID: "47d32cc6-fe53-440f-9aa1-053666e47df2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:41:30.039199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.039111    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7" pod "47d32cc6-fe53-440f-9aa1-053666e47df2" (UID: "47d32cc6-fe53-440f-9aa1-053666e47df2"). InnerVolumeSpecName "kube-api-access-krzl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:41:30.138254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.138228    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzl7\" (UniqueName: \"kubernetes.io/projected/47d32cc6-fe53-440f-9aa1-053666e47df2-kube-api-access-krzl7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:41:30.138254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.138252    5065 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d32cc6-fe53-440f-9aa1-053666e47df2-metrics-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:41:30.701249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.701204    5065 generic.go:356] "Generic (PLEG): container finished" podID="47d32cc6-fe53-440f-9aa1-053666e47df2" containerID="8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337" exitCode=0
Oct 30 18:41:30.701645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.701272    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27"
Oct 30 18:41:30.701645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.701297    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" event={"ID":"47d32cc6-fe53-440f-9aa1-053666e47df2","Type":"ContainerDied","Data":"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"}
Oct 30 18:41:30.701645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.701361    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6kn27" event={"ID":"47d32cc6-fe53-440f-9aa1-053666e47df2","Type":"ContainerDied","Data":"afdb7a5de1b05ef6e4cf98d80e21231b6b69d7046b7565b2156031104062092b"}
Oct 30 18:41:30.701645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.701385    5065 scope.go:122] "RemoveContainer" containerID="8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"
Oct 30 18:41:30.713955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.713930    5065 scope.go:122] "RemoveContainer" containerID="8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"
Oct 30 18:41:30.714297 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:41:30.714278    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337\": container with ID starting with 8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337 not found: ID does not exist" containerID="8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"
Oct 30 18:41:30.714369 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.714306    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337"} err="failed to get container status \"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337\": rpc error: code = NotFound desc = could not find container \"8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337\": container with ID starting with 8044cfcab8bcf94740e3629efb17eb790ff5939e8286b91fc30140ca77f6b337 not found: ID does not exist"
Oct 30 18:41:30.732826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.732796    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:30.739518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:30.739490    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6kn27"]
Oct 30 18:41:31.887521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:31.887492    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d32cc6-fe53-440f-9aa1-053666e47df2" path="/var/lib/kubelet/pods/47d32cc6-fe53-440f-9aa1-053666e47df2/volumes"
Oct 30 18:41:40.059880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.059850    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"]
Oct 30 18:41:40.060295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.060281    5065 cpu_manager.go:414] "containerMap: removing container" podUID="47d32cc6-fe53-440f-9aa1-053666e47df2" containerName="manager"
Oct 30 18:41:40.060338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.060296    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d32cc6-fe53-440f-9aa1-053666e47df2" containerName="manager"
Oct 30 18:41:40.060383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.060374    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d32cc6-fe53-440f-9aa1-053666e47df2" containerName="manager"
Oct 30 18:41:40.068531 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.068506    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.074661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.074637    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"]
Oct 30 18:41:40.077491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.077470    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 18:41:40.077814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.077484    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:41:40.077814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.077509    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zfsfz\""
Oct 30 18:41:40.077975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.077552    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:41:40.222860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.222815    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkh6\" (UniqueName: \"kubernetes.io/projected/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-kube-api-access-qkkh6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.223032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.222936    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.223032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.222968    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.223032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.223029    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.223187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.223101    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.223187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.223137    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324126    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkh6\" (UniqueName: \"kubernetes.io/projected/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-kube-api-access-qkkh6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324224    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324273    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324312    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324353    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324708    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.324868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.324813    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.326647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.326620    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.326956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.326937    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.337913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.337883    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkh6\" (UniqueName: \"kubernetes.io/projected/b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a-kube-api-access-qkkh6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tss5n\" (UID: \"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.384035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.384013    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:40.668656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.668632    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"]
Oct 30 18:41:40.671222 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:40.671187    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bcbabc_ea43_41cc_b40d_f0cfefae6f0a.slice/crio-3a53b33d1f1c6cf07863810db067f06bf13c508221db983b3b459a9bf7802d58 WatchSource:0}: Error finding container 3a53b33d1f1c6cf07863810db067f06bf13c508221db983b3b459a9bf7802d58: Status 404 returned error can't find the container with id 3a53b33d1f1c6cf07863810db067f06bf13c508221db983b3b459a9bf7802d58
Oct 30 18:41:40.762484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.762449    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n" event={"ID":"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a","Type":"ContainerStarted","Data":"3a53b33d1f1c6cf07863810db067f06bf13c508221db983b3b459a9bf7802d58"}
Oct 30 18:41:40.938174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.937238    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"]
Oct 30 18:41:40.946974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.946950    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:40.952803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.952780    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 18:41:40.953006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:40.952977    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"]
Oct 30 18:41:41.153755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.153712    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.153755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.153755    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.154332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.153880    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.154332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.153932    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9s7\" (UniqueName: \"kubernetes.io/projected/9ba78a05-9a5a-466a-bc96-0628a262466b-kube-api-access-2m9s7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.154332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.153965    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.154332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.154206    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba78a05-9a5a-466a-bc96-0628a262466b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256553    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256602    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256661    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256700    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9s7\" (UniqueName: \"kubernetes.io/projected/9ba78a05-9a5a-466a-bc96-0628a262466b-kube-api-access-2m9s7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256726    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.256870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.256769    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba78a05-9a5a-466a-bc96-0628a262466b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.257124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.257066    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.257492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.257159    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.257492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.257452    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.261044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.261013    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba78a05-9a5a-466a-bc96-0628a262466b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.263234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.263193    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba78a05-9a5a-466a-bc96-0628a262466b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.281129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.280711    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9s7\" (UniqueName: \"kubernetes.io/projected/9ba78a05-9a5a-466a-bc96-0628a262466b-kube-api-access-2m9s7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cjgq\" (UID: \"9ba78a05-9a5a-466a-bc96-0628a262466b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.566408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.566380    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:41.871417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:41.871276    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"]
Oct 30 18:41:41.872374 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:41.872348    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba78a05_9a5a_466a_bc96_0628a262466b.slice/crio-5b97a2d7c515db754b0c21943ce463f7588bd843f90828cb52361314e82b401f WatchSource:0}: Error finding container 5b97a2d7c515db754b0c21943ce463f7588bd843f90828cb52361314e82b401f: Status 404 returned error can't find the container with id 5b97a2d7c515db754b0c21943ce463f7588bd843f90828cb52361314e82b401f
Oct 30 18:41:42.798433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:42.798394    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n" event={"ID":"b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a","Type":"ContainerStarted","Data":"972d9820cb7a80807f5581f21b98d1384cfcd1347c843bcbab998e5d9ba31163"}
Oct 30 18:41:42.816728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:42.816650    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq" event={"ID":"9ba78a05-9a5a-466a-bc96-0628a262466b","Type":"ContainerStarted","Data":"5b97a2d7c515db754b0c21943ce463f7588bd843f90828cb52361314e82b401f"}
Oct 30 18:41:42.835523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:42.833192    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n" podStartSLOduration=1.022434778 podStartE2EDuration="2.833174534s" totalImagesPullingTime="1.810739756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:42.832305407 +0000 UTC m=+523.095770355" watchObservedRunningTime="2026-10-30 18:41:42.833174534 +0000 UTC m=+523.096639483"
Oct 30 18:41:43.385185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:43.384342    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:43.826601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:43.826558    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq" event={"ID":"9ba78a05-9a5a-466a-bc96-0628a262466b","Type":"ContainerStarted","Data":"9da6ccf1117d5dcdba1ff11510bf70f6d63aa35acd21cb6284712753bffaa999"}
Oct 30 18:41:43.849585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:43.849525    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq" podStartSLOduration=2.713910496 podStartE2EDuration="3.84950947s" totalImagesPullingTime="1.135598974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:43.849238516 +0000 UTC m=+524.112703481" watchObservedRunningTime="2026-10-30 18:41:43.84950947 +0000 UTC m=+524.112974418"
Oct 30 18:41:44.567126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.567065    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:44.786667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.786619    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"]
Oct 30 18:41:44.796281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.796255    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.801380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.801337    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"]
Oct 30 18:41:44.802256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.802233    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 18:41:44.822927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.822870    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.822927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.822903    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4pc\" (UniqueName: \"kubernetes.io/projected/dfeaf827-b7cd-4368-ae2c-b553d38301d0-kube-api-access-vv4pc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.823133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.822940    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.823181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.823148    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.823272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.823254    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.823311 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.823280    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.923877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.923848    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.923932    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.923957    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924024    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924099    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4pc\" (UniqueName: \"kubernetes.io/projected/dfeaf827-b7cd-4368-ae2c-b553d38301d0-kube-api-access-vv4pc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924332    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924376    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924465    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.924696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.924476    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.927014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.926973    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf827-b7cd-4368-ae2c-b553d38301d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.927743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.927717    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfeaf827-b7cd-4368-ae2c-b553d38301d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:44.938897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:44.938873    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4pc\" (UniqueName: \"kubernetes.io/projected/dfeaf827-b7cd-4368-ae2c-b553d38301d0-kube-api-access-vv4pc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6\" (UID: \"dfeaf827-b7cd-4368-ae2c-b553d38301d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:45.117603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:45.117537    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:45.401190 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:45.401163    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeaf827_b7cd_4368_ae2c_b553d38301d0.slice/crio-d368cbe3f792e35b2d09f61708fb26ac535f7a1e98a4c680b2c3a48136ee27a7 WatchSource:0}: Error finding container d368cbe3f792e35b2d09f61708fb26ac535f7a1e98a4c680b2c3a48136ee27a7: Status 404 returned error can't find the container with id d368cbe3f792e35b2d09f61708fb26ac535f7a1e98a4c680b2c3a48136ee27a7
Oct 30 18:41:45.401667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:45.401642    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"]
Oct 30 18:41:45.853789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:45.853118    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6" event={"ID":"dfeaf827-b7cd-4368-ae2c-b553d38301d0","Type":"ContainerStarted","Data":"54f34462f4c68d8afe5a7711ccdcacf837cd0164871a73b9f94e8548ef06f9c4"}
Oct 30 18:41:45.853789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:45.853189    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6" event={"ID":"dfeaf827-b7cd-4368-ae2c-b553d38301d0","Type":"ContainerStarted","Data":"d368cbe3f792e35b2d09f61708fb26ac535f7a1e98a4c680b2c3a48136ee27a7"}
Oct 30 18:41:45.878569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:45.878514    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6" podStartSLOduration=1.547381025 podStartE2EDuration="1.878496862s" totalImagesPullingTime="331.115837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:45.876430936 +0000 UTC m=+526.139895911" watchObservedRunningTime="2026-10-30 18:41:45.878496862 +0000 UTC m=+526.141961809"
Oct 30 18:41:46.117954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:46.117904    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:48.974580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:48.974536    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"]
Oct 30 18:41:48.978469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:48.978445    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:48.999173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:48.998733    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 18:41:49.016922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.016660    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"]
Oct 30 18:41:49.064769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.064741    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.064918 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.064810    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.064918 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.064896    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.065033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.065014    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.065072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.065046    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkq9\" (UniqueName: \"kubernetes.io/projected/fa456d21-114b-4b81-93b9-64f94cd6e963-kube-api-access-nrkq9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.065141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.065073    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa456d21-114b-4b81-93b9-64f94cd6e963-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166377    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa456d21-114b-4b81-93b9-64f94cd6e963-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166470    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166519    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166659    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166749    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166785    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkq9\" (UniqueName: \"kubernetes.io/projected/fa456d21-114b-4b81-93b9-64f94cd6e963-kube-api-access-nrkq9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.166951 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166929    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.167049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.166972    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.167049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.167018    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.169005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.168983    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa456d21-114b-4b81-93b9-64f94cd6e963-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.169005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.168999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa456d21-114b-4b81-93b9-64f94cd6e963-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.182344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.182308    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkq9\" (UniqueName: \"kubernetes.io/projected/fa456d21-114b-4b81-93b9-64f94cd6e963-kube-api-access-nrkq9\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7\" (UID: \"fa456d21-114b-4b81-93b9-64f94cd6e963\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.307771 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.307736    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:49.597541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.597509    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"]
Oct 30 18:41:49.598868 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:41:49.598840    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa456d21_114b_4b81_93b9_64f94cd6e963.slice/crio-e0e91b4d2f1145d4bb8be2827c3ad42f61c3089a399bb575b313414c12b6af9e WatchSource:0}: Error finding container e0e91b4d2f1145d4bb8be2827c3ad42f61c3089a399bb575b313414c12b6af9e: Status 404 returned error can't find the container with id e0e91b4d2f1145d4bb8be2827c3ad42f61c3089a399bb575b313414c12b6af9e
Oct 30 18:41:49.879935 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:49.879850    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" event={"ID":"fa456d21-114b-4b81-93b9-64f94cd6e963","Type":"ContainerStarted","Data":"e0e91b4d2f1145d4bb8be2827c3ad42f61c3089a399bb575b313414c12b6af9e"}
Oct 30 18:41:50.384841 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:50.384792    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:50.404623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:50.404590    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:50.887152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:50.887116    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" event={"ID":"fa456d21-114b-4b81-93b9-64f94cd6e963","Type":"ContainerStarted","Data":"093097cdc905e64b643e834d276cdbcdfec2826fdf7ab7fe77d77cf6c4260bc9"}
Oct 30 18:41:50.900918 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:50.900895    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tss5n"
Oct 30 18:41:50.914589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:50.914538    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podStartSLOduration=2.592764858 podStartE2EDuration="2.91452567s" totalImagesPullingTime="321.760812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:50.912424776 +0000 UTC m=+531.175889750" watchObservedRunningTime="2026-10-30 18:41:50.91452567 +0000 UTC m=+531.177990618"
Oct 30 18:41:51.308229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:51.308182    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:51.567191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:51.567070    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:51.580812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:51.580774    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:51.912666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:51.912576    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cjgq"
Oct 30 18:41:53.055855 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:41:53.055818    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:41:55.118419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:55.118387    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:55.131105 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:55.131062    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:55.930520 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:55.930488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6"
Oct 30 18:41:59.308592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:59.308556    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:59.321242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:59.321208    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:41:59.953292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:41:59.953269    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:42:02.880053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:02.880021    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bf474c468-bdpsm"]
Oct 30 18:42:02.885381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:02.885362    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:02.896131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:02.896107    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf474c468-bdpsm"]
Oct 30 18:42:02.987145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:02.987114    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfkk\" (UniqueName: \"kubernetes.io/projected/34502ce6-c71e-47ad-b608-7b50a8214301-kube-api-access-pnfkk\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:02.987281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:02.987259    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34502ce6-c71e-47ad-b608-7b50a8214301-tls-cert\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.088395 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.088365    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34502ce6-c71e-47ad-b608-7b50a8214301-tls-cert\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.088525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.088415    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfkk\" (UniqueName: \"kubernetes.io/projected/34502ce6-c71e-47ad-b608-7b50a8214301-kube-api-access-pnfkk\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.090944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.090919    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34502ce6-c71e-47ad-b608-7b50a8214301-tls-cert\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.102014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.101987    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfkk\" (UniqueName: \"kubernetes.io/projected/34502ce6-c71e-47ad-b608-7b50a8214301-kube-api-access-pnfkk\") pod \"authorino-6bf474c468-bdpsm\" (UID: \"34502ce6-c71e-47ad-b608-7b50a8214301\") " pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.202273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.202188    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf474c468-bdpsm"
Oct 30 18:42:03.482059 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.482034    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf474c468-bdpsm"]
Oct 30 18:42:03.483283 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:03.483253    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34502ce6_c71e_47ad_b608_7b50a8214301.slice/crio-c2851aec0b226bbc2fffb82e95ffc1a3063e6aa4d57494c6c5ff444a9b7ceaf8 WatchSource:0}: Error finding container c2851aec0b226bbc2fffb82e95ffc1a3063e6aa4d57494c6c5ff444a9b7ceaf8: Status 404 returned error can't find the container with id c2851aec0b226bbc2fffb82e95ffc1a3063e6aa4d57494c6c5ff444a9b7ceaf8
Oct 30 18:42:03.963879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:03.963847    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf474c468-bdpsm" event={"ID":"34502ce6-c71e-47ad-b608-7b50a8214301","Type":"ContainerStarted","Data":"c2851aec0b226bbc2fffb82e95ffc1a3063e6aa4d57494c6c5ff444a9b7ceaf8"}
Oct 30 18:42:04.973629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:04.973594    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf474c468-bdpsm" event={"ID":"34502ce6-c71e-47ad-b608-7b50a8214301","Type":"ContainerStarted","Data":"7e756ec4203a291ec2b9d1a66c173a4765af257876830d21f52a7215aa333f1c"}
Oct 30 18:42:04.998042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:04.997990    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bf474c468-bdpsm" podStartSLOduration=2.381360443 podStartE2EDuration="2.997974054s" totalImagesPullingTime="616.613611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:04.996430641 +0000 UTC m=+545.259895591" watchObservedRunningTime="2026-10-30 18:42:04.997974054 +0000 UTC m=+545.261439004"
Oct 30 18:42:05.028543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.028507    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:42:05.028800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.028752    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" podUID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" containerName="authorino" containerID="cri-o://e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac" gracePeriod=30
Oct 30 18:42:05.320081 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.320059    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:42:05.413149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.413112    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\" (UniqueName: \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\") pod \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") "
Oct 30 18:42:05.413389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.413222    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\" (UniqueName: \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\") pod \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\" (UID: \"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc\") "
Oct 30 18:42:05.416042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.416005    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd" pod "74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" (UID: "74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc"). InnerVolumeSpecName "kube-api-access-vs7sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:05.426943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.426917    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert" pod "74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" (UID: "74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:42:05.515232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.515182    5065 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-tls-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:05.515232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.515225    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7sd\" (UniqueName: \"kubernetes.io/projected/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc-kube-api-access-vs7sd\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:05.981592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.981559    5065 generic.go:356] "Generic (PLEG): container finished" podID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" containerID="e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac" exitCode=0
Oct 30 18:42:05.982028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.981622    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7cf978c5-c2scl"
Oct 30 18:42:05.982028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.981636    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" event={"ID":"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc","Type":"ContainerDied","Data":"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"}
Oct 30 18:42:05.982028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.981682    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d7cf978c5-c2scl" event={"ID":"74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc","Type":"ContainerDied","Data":"e1a482530c09b9776d7ab3a5833d8daea96e94bcf6187b9a1a58f07b33361ca6"}
Oct 30 18:42:05.982028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.981704    5065 scope.go:122] "RemoveContainer" containerID="e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"
Oct 30 18:42:05.996262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.996240    5065 scope.go:122] "RemoveContainer" containerID="e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"
Oct 30 18:42:05.996654 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:42:05.996629    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac\": container with ID starting with e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac not found: ID does not exist" containerID="e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"
Oct 30 18:42:05.996855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:05.996822    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac"} err="failed to get container status \"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac\": rpc error: code = NotFound desc = could not find container \"e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac\": container with ID starting with e8c64b497212e9788886671828791adde6dcfc038e98ce011ceb25ceedb639ac not found: ID does not exist"
Oct 30 18:42:06.011789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.011734    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:42:06.014706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.014683    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d7cf978c5-c2scl"]
Oct 30 18:42:06.147369 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.147342    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"]
Oct 30 18:42:06.148263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.148233    5065 cpu_manager.go:414] "containerMap: removing container" podUID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" containerName="authorino"
Oct 30 18:42:06.148263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.148262    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" containerName="authorino"
Oct 30 18:42:06.148451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.148413    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" containerName="authorino"
Oct 30 18:42:06.153847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.153785    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.171780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.171706    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"]
Oct 30 18:42:06.223384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223348    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223402    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223542    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223701 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223598    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea311f74-4221-4d7a-a3da-bc4a30848044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223701 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223644    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223760    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223835    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223866    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtql\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-kube-api-access-pjtql\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.223942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.223927    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.324876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.324839    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.324892    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea311f74-4221-4d7a-a3da-bc4a30848044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.324922    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.324999    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325056    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325078    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtql\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-kube-api-access-pjtql\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325388    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325453    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325523    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325564    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325594    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325620    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.325803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.325765    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea311f74-4221-4d7a-a3da-bc4a30848044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.326416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.326389    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.328274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.328249    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.328605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.328583    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.344427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.344395    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtql\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-kube-api-access-pjtql\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.347693 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.347652    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea311f74-4221-4d7a-a3da-bc4a30848044-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-p8smm\" (UID: \"ea311f74-4221-4d7a-a3da-bc4a30848044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.468043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.467942    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:06.815188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.815155    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"]
Oct 30 18:42:06.818717 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:06.818689    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea311f74_4221_4d7a_a3da_bc4a30848044.slice/crio-22b0acedf242ce0799ef6f65b73fa8c964c7b7c08db344a3f8fc0492b1f21496 WatchSource:0}: Error finding container 22b0acedf242ce0799ef6f65b73fa8c964c7b7c08db344a3f8fc0492b1f21496: Status 404 returned error can't find the container with id 22b0acedf242ce0799ef6f65b73fa8c964c7b7c08db344a3f8fc0492b1f21496
Oct 30 18:42:06.823937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.823897    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:06.824014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.823992    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:06.824049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.824038    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:06.993791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.993750    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" event={"ID":"ea311f74-4221-4d7a-a3da-bc4a30848044","Type":"ContainerStarted","Data":"d3d33b27e761abd6bca8418c3e0835e75e06121bb1a9baddc031c35ccb2e5fb8"}
Oct 30 18:42:06.994297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:06.993816    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" event={"ID":"ea311f74-4221-4d7a-a3da-bc4a30848044","Type":"ContainerStarted","Data":"22b0acedf242ce0799ef6f65b73fa8c964c7b7c08db344a3f8fc0492b1f21496"}
Oct 30 18:42:07.025220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:07.025157    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podStartSLOduration=1.025137022 podStartE2EDuration="1.025137022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:07.022460638 +0000 UTC m=+547.285925586" watchObservedRunningTime="2026-10-30 18:42:07.025137022 +0000 UTC m=+547.288601971"
Oct 30 18:42:07.468336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:07.468298    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:07.470197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:07.470168    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:07.470360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:07.470220    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:07.912579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:07.912544    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc" path="/var/lib/kubelet/pods/74ed99ae-c6cc-4da9-b82e-43eb38f0d9cc/volumes"
Oct 30 18:42:08.468653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:08.468618    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:08.469016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:08.468682    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:09.468919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:09.468885    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:09.469314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:09.468940    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:10.468709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:10.468665    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:10.468865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:10.468781    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:11.468940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:11.468903    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:11.469433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:11.468961    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:12.469111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:12.468524    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:12.469111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:12.468588    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:13.468657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:13.468625    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:42:13.468823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:13.468690    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 18:42:15.248415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:15.248374    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:15.248885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:15.248829    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:15.249757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:15.249735    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm"
Oct 30 18:42:21.227168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.227136    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"]
Oct 30 18:42:21.232687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.232667    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.264500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.264474    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"]
Oct 30 18:42:21.415798 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.415762    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1f838b-93cf-425f-b9ac-232488fca33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.415946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.415816    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.415946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.415880    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.415946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.415929    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.416053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.416027    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.416120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.416077    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.416174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.416140    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.416213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.416175    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.416280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.416265    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlcf\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-kube-api-access-mnlcf\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517224    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517265    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517289    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517307    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517335    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlcf\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-kube-api-access-mnlcf\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517373    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1f838b-93cf-425f-b9ac-232488fca33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517417    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517436    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517497    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517757    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517776    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517840    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.517979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.517963    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.518408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.518382    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1f838b-93cf-425f-b9ac-232488fca33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.519757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.519732    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.519854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.519737    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.531422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.531395    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.531598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.531570    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlcf\" (UniqueName: \"kubernetes.io/projected/ca1f838b-93cf-425f-b9ac-232488fca33e-kube-api-access-mnlcf\") pod \"maas-default-gateway-openshift-default-855f9574cf-v8w6k\" (UID: \"ca1f838b-93cf-425f-b9ac-232488fca33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.548432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.548412    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:21.839176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.839148    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"]
Oct 30 18:42:21.839294 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:21.839196    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1f838b_93cf_425f_b9ac_232488fca33e.slice/crio-0008148c0d8bfbd50509eb04db251e43e6ec4b317eb82f42e207beb4e5238d8e WatchSource:0}: Error finding container 0008148c0d8bfbd50509eb04db251e43e6ec4b317eb82f42e207beb4e5238d8e: Status 404 returned error can't find the container with id 0008148c0d8bfbd50509eb04db251e43e6ec4b317eb82f42e207beb4e5238d8e
Oct 30 18:42:21.841487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.841458    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:21.841566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.841520    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:21.841566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:21.841548    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:22.109854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.109775    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" event={"ID":"ca1f838b-93cf-425f-b9ac-232488fca33e","Type":"ContainerStarted","Data":"45ff9ed7d82fd765dd9e2b5c4d382992be800ed6f74b3195c776f1c99246f60e"}
Oct 30 18:42:22.109854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.109838    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" event={"ID":"ca1f838b-93cf-425f-b9ac-232488fca33e","Type":"ContainerStarted","Data":"0008148c0d8bfbd50509eb04db251e43e6ec4b317eb82f42e207beb4e5238d8e"}
Oct 30 18:42:22.142502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.142435    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podStartSLOduration=1.142415096 podStartE2EDuration="1.142415096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:22.138819015 +0000 UTC m=+562.402283973" watchObservedRunningTime="2026-10-30 18:42:22.142415096 +0000 UTC m=+562.405880042"
Oct 30 18:42:22.549324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.549291    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:22.550522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.550501    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:22.550584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:22.550547    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:23.057210 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:42:23.057172    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:42:23.549729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:23.549702    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:23.550172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:23.549753    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:24.548880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:24.548846    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:24.549054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:24.548911    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:25.548842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:25.548806    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:25.549315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:25.548876    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:26.548875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:26.548840    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:26.549307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:26.548905    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:27.549663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:27.549628    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:27.550202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:27.549701    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:28.549327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:28.549288    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:42:28.549496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:28.549348    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 18:42:30.002192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:30.002153    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:30.002833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:30.002811    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:30.007623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:30.007604    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k"
Oct 30 18:42:35.275386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.275346    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:42:35.281488 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.281464    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.296246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.296215    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:42:35.296522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.296497    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7289525-openshift-default-dockercfg-g9p8h\""
Oct 30 18:42:35.332682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332653    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332689    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332724    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nng\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332867    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332924    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.332969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332948    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.333061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.332985    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.333143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.333131    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.425518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.425488    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:42:35.429757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.429737    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.434265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434246    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434332    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434373    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434394    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434465    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434514    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nng\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434554    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434591    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434620    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.434634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434625    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.435126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.434980    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.435126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.435032    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.435126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.435103    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.435635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.435608    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.437210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.437019    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.437387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.437361    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.448518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.448499    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f5768279-openshift-default-dockercfg-fn9fd\""
Oct 30 18:42:35.455982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.455962    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:42:35.464846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.464824    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.465296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.465271    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nng\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\") pod \"e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") " pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535270    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535356    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535398    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535446    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535496    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535604    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535655    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kzc\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535688    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.536037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.535731    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.602277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.602236    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:35.637026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.636990    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637168    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kzc\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637225    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637266    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637350    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637454    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637500    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637561    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.637683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.637629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.638275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.638251    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.639471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.639252    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.641012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.640744    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.641153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.641067    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.643594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.643458    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.645282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.645252    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.648401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.647152    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.657870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.657841    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kzc\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.664507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.664265    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\") pod \"e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") " pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.753682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.753286    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:35.914774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.914738    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:42:35.926279 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:35.919219    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b45917d_1145_449f_a26f_4fccb2e09057.slice/crio-0d06997447c51dbc0d73250b15c7103c521c2b123a7073667ebff1f3148c7387 WatchSource:0}: Error finding container 0d06997447c51dbc0d73250b15c7103c521c2b123a7073667ebff1f3148c7387: Status 404 returned error can't find the container with id 0d06997447c51dbc0d73250b15c7103c521c2b123a7073667ebff1f3148c7387
Oct 30 18:42:35.926279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.922518    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:35.926279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.922596    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:35.926279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:35.922634    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:36.065929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.065895    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:42:36.070154 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:36.070124    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378b82f2_e4bb_4afc_8e97_4d0ca8c34102.slice/crio-8063572024cf862425f80ac03002a7daeab59c53e164226438a1665449bb561c WatchSource:0}: Error finding container 8063572024cf862425f80ac03002a7daeab59c53e164226438a1665449bb561c: Status 404 returned error can't find the container with id 8063572024cf862425f80ac03002a7daeab59c53e164226438a1665449bb561c
Oct 30 18:42:36.072691 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.072653    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:36.072767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.072743    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:36.072804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.072787    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:36.188669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.188631    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerStarted","Data":"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"}
Oct 30 18:42:36.188840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.188702    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerStarted","Data":"0d06997447c51dbc0d73250b15c7103c521c2b123a7073667ebff1f3148c7387"}
Oct 30 18:42:36.191682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.191652    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerStarted","Data":"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"}
Oct 30 18:42:36.191825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.191727    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerStarted","Data":"8063572024cf862425f80ac03002a7daeab59c53e164226438a1665449bb561c"}
Oct 30 18:42:36.221878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.221819    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podStartSLOduration=1.221801471 podStartE2EDuration="1.221801471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:36.218399766 +0000 UTC m=+576.481864721" watchObservedRunningTime="2026-10-30 18:42:36.221801471 +0000 UTC m=+576.485266421"
Oct 30 18:42:36.261576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.261516    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podStartSLOduration=1.2614962410000001 podStartE2EDuration="1.261496241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:36.25805472 +0000 UTC m=+576.521519710" watchObservedRunningTime="2026-10-30 18:42:36.261496241 +0000 UTC m=+576.524961196"
Oct 30 18:42:36.602558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.602526    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:36.608660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.608636    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:36.753929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:36.753900    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:37.198246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:37.198211    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:37.199105 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:37.199071    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:42:37.759667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:37.759636    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:38.204035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:38.203955    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:38.205355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:38.205327    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:39.236061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.236030    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:42:39.239998 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.239982    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.251188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.251169    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f5768279\""
Oct 30 18:42:39.261199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.261174    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:42:39.274579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.274552    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.274871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.274844    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhvk\" (UniqueName: \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.291736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.291705    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:42:39.295289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.295274    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.312278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.312260    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f5768279-dockercfg-4zwqh\""
Oct 30 18:42:39.323502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.323479    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:42:39.377109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.376643    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.379663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.377502    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhvk\" (UniqueName: \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.379663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.377678    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bkj\" (UniqueName: \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.379663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.378054    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.379663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.379291    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.395694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.395671    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhvk\" (UniqueName: \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\") pod \"payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.480590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.480563    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.480756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.480634    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bkj\" (UniqueName: \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.481336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.481312    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.494636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.494569    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bkj\" (UniqueName: \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\") pod \"payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") " pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.553656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.553633    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:39.607670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.607273    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:39.844759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.844737    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:42:39.848212 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:39.848184    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174b1bba_bcf1_4f82_b581_1602bd8b610b.slice/crio-5ba69f108db959cc96aff57c76ba0b12edcb4266e5207cd517165011b3156060 WatchSource:0}: Error finding container 5ba69f108db959cc96aff57c76ba0b12edcb4266e5207cd517165011b3156060: Status 404 returned error can't find the container with id 5ba69f108db959cc96aff57c76ba0b12edcb4266e5207cd517165011b3156060
Oct 30 18:42:39.915408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:39.915381    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:42:39.919561 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:39.919532    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdb8d01_d52e_4791_b227_5ce437d298c7.slice/crio-c2be9b7a879e99539592313db9431f74e1a44ca15214b466397599a118d70005 WatchSource:0}: Error finding container c2be9b7a879e99539592313db9431f74e1a44ca15214b466397599a118d70005: Status 404 returned error can't find the container with id c2be9b7a879e99539592313db9431f74e1a44ca15214b466397599a118d70005
Oct 30 18:42:40.003040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.003014    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:42:40.006833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.006813    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.023339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.023318    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d7289525-dockercfg-4srkk\""
Oct 30 18:42:40.024417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.024390    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:42:40.065007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.064982    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7289525\""
Oct 30 18:42:40.094624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.086966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.094624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.087015    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6nf\" (UniqueName: \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.187869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.187825    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.187869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.187869    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6nf\" (UniqueName: \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.188604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.188579    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.205153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.205123    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6nf\" (UniqueName: \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\") pod \"payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") " pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.224666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.224618    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" event={"ID":"174b1bba-bcf1-4f82-b581-1602bd8b610b","Type":"ContainerStarted","Data":"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"}
Oct 30 18:42:40.224852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.224740    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" event={"ID":"174b1bba-bcf1-4f82-b581-1602bd8b610b","Type":"ContainerStarted","Data":"5ba69f108db959cc96aff57c76ba0b12edcb4266e5207cd517165011b3156060"}
Oct 30 18:42:40.224852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.224780    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:40.226380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.226360    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" event={"ID":"8bdb8d01-d52e-4791-b227-5ce437d298c7","Type":"ContainerStarted","Data":"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"}
Oct 30 18:42:40.226505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.226403    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:40.226505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.226425    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" event={"ID":"8bdb8d01-d52e-4791-b227-5ce437d298c7","Type":"ContainerStarted","Data":"c2be9b7a879e99539592313db9431f74e1a44ca15214b466397599a118d70005"}
Oct 30 18:42:40.248724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.248684    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" podStartSLOduration=1.248672139 podStartE2EDuration="1.248672139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:40.246676848 +0000 UTC m=+580.510141826" watchObservedRunningTime="2026-10-30 18:42:40.248672139 +0000 UTC m=+580.512137088"
Oct 30 18:42:40.277518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.277453    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" podStartSLOduration=1.277435145 podStartE2EDuration="1.277435145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:40.276337932 +0000 UTC m=+580.539802894" watchObservedRunningTime="2026-10-30 18:42:40.277435145 +0000 UTC m=+580.540900094"
Oct 30 18:42:40.321633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.321557    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:40.621729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:40.621646    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:42:41.238175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:41.238143    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" event={"ID":"9de91640-3a67-4722-be4b-1d281c8b63f9","Type":"ContainerStarted","Data":"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"}
Oct 30 18:42:41.238366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:41.238200    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" event={"ID":"9de91640-3a67-4722-be4b-1d281c8b63f9","Type":"ContainerStarted","Data":"895390fcc34e96674072cfaf8bfcfc2112ca97ce76090eea1f68cdfe0c486dac"}
Oct 30 18:42:41.238518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:41.238496    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:41.262827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:41.262782    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" podStartSLOduration=2.262769211 podStartE2EDuration="2.262769211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.261799033 +0000 UTC m=+581.525264001" watchObservedRunningTime="2026-10-30 18:42:41.262769211 +0000 UTC m=+581.526234192"
Oct 30 18:42:42.155168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.155139    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:42:42.158865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.158848    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.171745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.171717    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:42:42.172073 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.171914    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9aded813\""
Oct 30 18:42:42.172542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.172521    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9aded813-dockercfg-7xmlt\""
Oct 30 18:42:42.219581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.219550    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7f6\" (UniqueName: \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.219730 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.219618    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.321206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.321168    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7f6\" (UniqueName: \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.321640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.321321    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.322005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.321984    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.334973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.334945    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7f6\" (UniqueName: \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\") pod \"payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.473437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.473350    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:42.765473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:42.765445    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:42:42.766455 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:42.766429    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a408941_edc5_4859_a647_1bc143c67952.slice/crio-e3443ae78571e836a83e7b758abe5a3ed142a02fe5628b0a7888fe9ce693f11d WatchSource:0}: Error finding container e3443ae78571e836a83e7b758abe5a3ed142a02fe5628b0a7888fe9ce693f11d: Status 404 returned error can't find the container with id e3443ae78571e836a83e7b758abe5a3ed142a02fe5628b0a7888fe9ce693f11d
Oct 30 18:42:43.256625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:43.256531    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" event={"ID":"7a408941-edc5-4859-a647-1bc143c67952","Type":"ContainerStarted","Data":"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"}
Oct 30 18:42:43.256768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:43.256632    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:43.256768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:43.256655    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" event={"ID":"7a408941-edc5-4859-a647-1bc143c67952","Type":"ContainerStarted","Data":"e3443ae78571e836a83e7b758abe5a3ed142a02fe5628b0a7888fe9ce693f11d"}
Oct 30 18:42:43.281962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:43.281904    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" podStartSLOduration=1.281886335 podStartE2EDuration="1.281886335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:43.27817656 +0000 UTC m=+583.541641535" watchObservedRunningTime="2026-10-30 18:42:43.281886335 +0000 UTC m=+583.545351285"
Oct 30 18:42:44.557462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.557430    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:42:44.561723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.561700    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.570147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.570126    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-72bc98fe-openshift-default-dockercfg-28qpr\""
Oct 30 18:42:44.615955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.615915    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:42:44.647754 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647726    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.647916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647765    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.647916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647787    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.647989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647936    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dd9\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.647989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647972    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.648053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.647997    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.648115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.648099    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.648153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.648138    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.648188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.648176    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749418    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749459    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749610    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dd9\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749670    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749716    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749776    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749815    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.749907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749887    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.749920    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.750014    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.750196    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.750348    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.750519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.750680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.750659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.751845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.751828    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.752850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.752830    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.767526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.767494    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dd9\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.767765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.767744    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\") pod \"e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") " pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:44.876261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:44.876186    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:45.213421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:45.213327    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:42:45.217497 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:45.217458    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4842e58_a7a7_415e_b12c_08e9c536915e.slice/crio-17f2ef5fde0aec1930b09b8a5387b81fc10af00415bb7514de12b2e717861738 WatchSource:0}: Error finding container 17f2ef5fde0aec1930b09b8a5387b81fc10af00415bb7514de12b2e717861738: Status 404 returned error can't find the container with id 17f2ef5fde0aec1930b09b8a5387b81fc10af00415bb7514de12b2e717861738
Oct 30 18:42:45.219976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:45.219946    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:45.220114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:45.220011    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:45.220114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:45.220040    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:45.269318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:45.269287    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerStarted","Data":"17f2ef5fde0aec1930b09b8a5387b81fc10af00415bb7514de12b2e717861738"}
Oct 30 18:42:46.277366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:46.277333    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerStarted","Data":"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"}
Oct 30 18:42:46.324953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:46.324903    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podStartSLOduration=2.324888314 podStartE2EDuration="2.324888314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:46.322476377 +0000 UTC m=+586.585941344" watchObservedRunningTime="2026-10-30 18:42:46.324888314 +0000 UTC m=+586.588353258"
Oct 30 18:42:46.877266 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:46.877228    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:46.889111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:46.888221    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:47.302109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:47.301834    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:47.314108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:47.309316    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:42:49.271977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.271947    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:42:49.276225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.276196    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.303446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.303414    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-72bc98fe\""
Oct 30 18:42:49.330113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.305822    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-72bc98fe-dockercfg-h8998\""
Oct 30 18:42:49.350209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.350165    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:42:49.355499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.355223    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:42:49.359925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.359902    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:42:49.360362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.360345    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.370762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.370730    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.371017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.371000    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sz98\" (UniqueName: \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.472695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.472646    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sz98\" (UniqueName: \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.473380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.473020    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjx5\" (UniqueName: \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.473540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.473522    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.473697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.473682    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.474817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.474717    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.518990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.518958    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sz98\" (UniqueName: \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\") pod \"payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") " pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.575971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.575892    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjx5\" (UniqueName: \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.575971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.575930    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.576685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.576659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.611158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.611067    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjx5\" (UniqueName: \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\") pod \"payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:49.670710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.670668    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:49.716099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:49.716043    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:50.016372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.015927    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:42:50.019973 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:50.019941    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad90645b_6109_4282_a3ad_d33657aa5705.slice/crio-4fd9276196fff905e1398a358bbf21bc857b5891e3d098648288d2d3957a9951 WatchSource:0}: Error finding container 4fd9276196fff905e1398a358bbf21bc857b5891e3d098648288d2d3957a9951: Status 404 returned error can't find the container with id 4fd9276196fff905e1398a358bbf21bc857b5891e3d098648288d2d3957a9951
Oct 30 18:42:50.063329 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.063300    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:42:50.363383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.363342    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" event={"ID":"ad90645b-6109-4282-a3ad-d33657aa5705","Type":"ContainerStarted","Data":"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"}
Oct 30 18:42:50.363816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.363446    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:42:50.363816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.363474    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" event={"ID":"ad90645b-6109-4282-a3ad-d33657aa5705","Type":"ContainerStarted","Data":"4fd9276196fff905e1398a358bbf21bc857b5891e3d098648288d2d3957a9951"}
Oct 30 18:42:50.365278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.365246    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" event={"ID":"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5","Type":"ContainerStarted","Data":"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"}
Oct 30 18:42:50.365412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.365307    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" event={"ID":"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5","Type":"ContainerStarted","Data":"e08b930938c9a4cb6eebe5f3a952590a3cb4558550e7efc6ea6a4157111da4e7"}
Oct 30 18:42:50.365412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.365391    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:42:50.406204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.405372    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" podStartSLOduration=1.405357461 podStartE2EDuration="1.405357461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:50.401471973 +0000 UTC m=+590.664936949" watchObservedRunningTime="2026-10-30 18:42:50.405357461 +0000 UTC m=+590.668822410"
Oct 30 18:42:50.428367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:50.428323    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" podStartSLOduration=1.428308956 podStartE2EDuration="1.428308956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:50.426210063 +0000 UTC m=+590.689675027" watchObservedRunningTime="2026-10-30 18:42:50.428308956 +0000 UTC m=+590.691773945"
Oct 30 18:42:51.242233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:51.242199    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:42:51.242983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:51.242958    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:42:52.252682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:52.252655    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:42:53.058323 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:42:53.058272    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:42:53.733692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.733647    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"]
Oct 30 18:42:53.739134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.739107    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.745517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.744603    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:42:53.745749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.745014    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:42:53.747152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.747133    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"]
Oct 30 18:42:53.781240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.781125    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcv2\" (UniqueName: \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.781408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.781343    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.781513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.781492    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.882863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.882824    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcv2\" (UniqueName: \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.883038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.882927    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.883038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.882964    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.883769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.883736    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.885736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.885684    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:53.901432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:53.901408    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcv2\" (UniqueName: \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\") pod \"maas-api-revoke-keys-e2e-ait-f5768279-7jc8f\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:54.061012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.060973    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:42:54.249280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.249247    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:42:54.253263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.249325    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:42:54.264787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.264766    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:42:54.352241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.350849    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"]
Oct 30 18:42:54.354255 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:42:54.354215    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cabe4e2_1eec_47c0_9f8a_e94837963c25.slice/crio-03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d WatchSource:0}: Error finding container 03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d: Status 404 returned error can't find the container with id 03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d
Oct 30 18:42:54.407766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.407729    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f" event={"ID":"0cabe4e2-1eec-47c0-9f8a-e94837963c25","Type":"ContainerStarted","Data":"03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d"}
Oct 30 18:42:54.409419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.409396    5065 generic.go:356] "Generic (PLEG): container finished" podID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerID="2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e" exitCode=0
Oct 30 18:42:54.409534 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.409472    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerDied","Data":"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"}
Oct 30 18:42:54.409876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.409859    5065 scope.go:122] "RemoveContainer" containerID="2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"
Oct 30 18:42:54.411218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.411193    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:54.411306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.411257    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:54.411306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:54.411289    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:55.420550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:55.420509    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerStarted","Data":"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"}
Oct 30 18:42:55.754706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:55.754619    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:42:55.754928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:55.754843    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:42:55.754928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:55.754901    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:42:56.754931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:56.754897    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:42:56.755445 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:56.754945    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:42:57.754762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:57.754725    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:42:57.754961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:57.754789    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:42:58.754550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:58.754511    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:42:58.754744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:58.754592    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:42:59.178939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.178910    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"]
Oct 30 18:42:59.222667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.222639    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"]
Oct 30 18:42:59.222957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.222941    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.264920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.264890    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"]
Oct 30 18:42:59.272118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.271360    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.272355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.272332    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5clrx\" (UniqueName: \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.272727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.272476    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.272727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.272604    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.272924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.272903    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"]
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.373849    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.373907    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.373937    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trktf\" (UniqueName: \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.373966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.374007    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.374700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.374057    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5clrx\" (UniqueName: \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.375405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.375385    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.421704 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.420524    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.438156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.436996    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5clrx\" (UniqueName: \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\") pod \"maas-api-revoke-keys-e2e-migrate-78934afe-p468b\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.465456    5065 generic.go:356] "Generic (PLEG): container finished" podID="9b45917d-1145-449f-a26f-4fccb2e09057" containerID="540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4" exitCode=0
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.465528    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerDied","Data":"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"}
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.465875    5065 scope.go:122] "RemoveContainer" containerID="540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.467738    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.467819    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:59.470136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.467856    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:42:59.474686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.474647    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.474850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.474822    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.474934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.474895    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trktf\" (UniqueName: \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.476123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.476060    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.478316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.478289    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.560709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.560680    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trktf\" (UniqueName: \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\") pod \"maas-api-revoke-keys-e2e-ait-d7289525-pc6n7\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.576126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.576068    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:42:59.657960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.657737    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:42:59.764172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.763719    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:42:59.764172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:42:59.763775    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:43:00.248162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.238949    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"]
Oct 30 18:43:00.261129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.261107    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:43:00.276116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.275688    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:43:00.289434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.289349    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:43:00.315886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.315862    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"]
Oct 30 18:43:00.356130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.354373    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:43:00.356130 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:00.354872    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89300a9e_0887_4e6b_8641_e093cd094c52.slice/crio-03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266 WatchSource:0}: Error finding container 03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266: Status 404 returned error can't find the container with id 03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266
Oct 30 18:43:00.480110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.477483    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerStarted","Data":"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"}
Oct 30 18:43:00.488109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.487045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7" event={"ID":"89300a9e-0887-4e6b-8641-e093cd094c52","Type":"ContainerStarted","Data":"03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266"}
Oct 30 18:43:00.509172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.495527    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b" event={"ID":"489389b0-0319-41b7-afcf-8ee8c92dbbec","Type":"ContainerStarted","Data":"9c9d9cea89f3b38eb76346ec9a2d81bf9c392a222fb749631926a0e79cdc1e31"}
Oct 30 18:43:00.605668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.605637    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:00.605992 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.605972    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:00.606183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.606149    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:00.757933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.757897    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:43:00.758148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.757984    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:43:00.874165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.874054    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"]
Oct 30 18:43:00.881327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.881304    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:00.916108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.915769    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"]
Oct 30 18:43:00.972232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.971434    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:00.972232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.971504    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:00.972232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:00.971698    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrg6d\" (UniqueName: \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.073064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.073026    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.075687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.073171    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrg6d\" (UniqueName: \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.075687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.073227    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.075687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.074036    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.078749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.078718    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.148163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.147375    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrg6d\" (UniqueName: \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\") pod \"maas-api-revoke-keys-e2e-ait-9aded813-cr7nw\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.273436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.273404    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:01.377946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.377917    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:43:01.378135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.378006    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:43:01.611116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.603549    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:01.611116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.603619    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:01.760535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.758863    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:43:01.760535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.760297    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:43:01.761253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:01.761159    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:43:02.044011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:02.043948    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"]
Oct 30 18:43:02.604976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:02.604469    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:02.604976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:02.604529    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:02.671046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:02.670994    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw" event={"ID":"b0bc661f-c2c7-40cc-90bc-7eb69438a598","Type":"ContainerStarted","Data":"a99bebb3150a210b82bed707fd3385e3f6c220cbebef38f7f5d80f685732a274"}
Oct 30 18:43:02.674672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:02.674174    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f" event={"ID":"0cabe4e2-1eec-47c0-9f8a-e94837963c25","Type":"ContainerStarted","Data":"04c06eed4aa74058af403a3285589a025645c5a7ad6e534dddffe92fc016e05d"}
Oct 30 18:43:03.605729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:03.605394    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:03.605729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:03.605442    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:03.708376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:03.708341    5065 generic.go:356] "Generic (PLEG): container finished" podID="0cabe4e2-1eec-47c0-9f8a-e94837963c25" containerID="04c06eed4aa74058af403a3285589a025645c5a7ad6e534dddffe92fc016e05d" exitCode=0
Oct 30 18:43:03.708560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:03.708425    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f" event={"ID":"0cabe4e2-1eec-47c0-9f8a-e94837963c25","Type":"ContainerDied","Data":"04c06eed4aa74058af403a3285589a025645c5a7ad6e534dddffe92fc016e05d"}
Oct 30 18:43:04.604511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.604475    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:04.604710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.604534    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:04.717879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.717839    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw" event={"ID":"b0bc661f-c2c7-40cc-90bc-7eb69438a598","Type":"ContainerStarted","Data":"c051e5a08db4cfc7b64c4f3d7ccee5bfc78c4175be557dc8cb260bf0b12918da"}
Oct 30 18:43:04.719674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.719645    5065 generic.go:356] "Generic (PLEG): container finished" podID="89300a9e-0887-4e6b-8641-e093cd094c52" containerID="d0a11033470d8ce7fa0d3f8a07d239aa8ca78be11e932135cd5460c95c06bc94" exitCode=0
Oct 30 18:43:04.719791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.719723    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7" event={"ID":"89300a9e-0887-4e6b-8641-e093cd094c52","Type":"ContainerDied","Data":"d0a11033470d8ce7fa0d3f8a07d239aa8ca78be11e932135cd5460c95c06bc94"}
Oct 30 18:43:04.721521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.721499    5065 generic.go:356] "Generic (PLEG): container finished" podID="489389b0-0319-41b7-afcf-8ee8c92dbbec" containerID="93ebffc1a56f5ca1162f17601040d59a78cff92b0094efd7d2f58f46252706f5" exitCode=0
Oct 30 18:43:04.721617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.721550    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b" event={"ID":"489389b0-0319-41b7-afcf-8ee8c92dbbec","Type":"ContainerDied","Data":"93ebffc1a56f5ca1162f17601040d59a78cff92b0094efd7d2f58f46252706f5"}
Oct 30 18:43:04.926227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:04.926200    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:43:05.011047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.011016    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\") pod \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") "
Oct 30 18:43:05.011258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.011237    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\" (UniqueName: \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\") pod \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") "
Oct 30 18:43:05.011339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.011278    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\") pod \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\" (UID: \"0cabe4e2-1eec-47c0-9f8a-e94837963c25\") "
Oct 30 18:43:05.011442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.011417    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca" pod "0cabe4e2-1eec-47c0-9f8a-e94837963c25" (UID: "0cabe4e2-1eec-47c0-9f8a-e94837963c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:05.011743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.011720    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:05.013779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.013754    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca" pod "0cabe4e2-1eec-47c0-9f8a-e94837963c25" (UID: "0cabe4e2-1eec-47c0-9f8a-e94837963c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:05.014259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.014244    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2" pod "0cabe4e2-1eec-47c0-9f8a-e94837963c25" (UID: "0cabe4e2-1eec-47c0-9f8a-e94837963c25"). InnerVolumeSpecName "kube-api-access-ghcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:05.112968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.112928    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcv2\" (UniqueName: \"kubernetes.io/projected/0cabe4e2-1eec-47c0-9f8a-e94837963c25-kube-api-access-ghcv2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:05.112968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.112965    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabe4e2-1eec-47c0-9f8a-e94837963c25-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:05.603914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.603444    5065 patch_prober.go:29] interesting pod/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:43:05.603914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.603506    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 18:43:05.729449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.729361    5065 generic.go:356] "Generic (PLEG): container finished" podID="b0bc661f-c2c7-40cc-90bc-7eb69438a598" containerID="c051e5a08db4cfc7b64c4f3d7ccee5bfc78c4175be557dc8cb260bf0b12918da" exitCode=0
Oct 30 18:43:05.729827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.729450    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw" event={"ID":"b0bc661f-c2c7-40cc-90bc-7eb69438a598","Type":"ContainerDied","Data":"c051e5a08db4cfc7b64c4f3d7ccee5bfc78c4175be557dc8cb260bf0b12918da"}
Oct 30 18:43:05.731167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.731127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f" event={"ID":"0cabe4e2-1eec-47c0-9f8a-e94837963c25","Type":"ContainerDied","Data":"03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d"}
Oct 30 18:43:05.731282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.731174    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03323d38480f8f129a7ef17c7c1bfd0ccc4313f2350ac96fd2d87ec0351b528d"
Oct 30 18:43:05.731282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.731136    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"
Oct 30 18:43:05.959409 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:05.959382    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:06.014776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.014749    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:43:06.016869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.016846    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:43:06.126170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126138    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\" (UniqueName: \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\") pod \"489389b0-0319-41b7-afcf-8ee8c92dbbec\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") "
Oct 30 18:43:06.126368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126186    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\") pod \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") "
Oct 30 18:43:06.126368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126247    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\") pod \"89300a9e-0887-4e6b-8641-e093cd094c52\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") "
Oct 30 18:43:06.126368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126313    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\" (UniqueName: \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\") pod \"89300a9e-0887-4e6b-8641-e093cd094c52\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") "
Oct 30 18:43:06.126368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126353    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\" (UniqueName: \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\") pod \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") "
Oct 30 18:43:06.126584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126453    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\") pod \"489389b0-0319-41b7-afcf-8ee8c92dbbec\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") "
Oct 30 18:43:06.126584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126486    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\") pod \"489389b0-0319-41b7-afcf-8ee8c92dbbec\" (UID: \"489389b0-0319-41b7-afcf-8ee8c92dbbec\") "
Oct 30 18:43:06.126584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126528    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\") pod \"89300a9e-0887-4e6b-8641-e093cd094c52\" (UID: \"89300a9e-0887-4e6b-8641-e093cd094c52\") "
Oct 30 18:43:06.126735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126587    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\") pod \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\" (UID: \"b0bc661f-c2c7-40cc-90bc-7eb69438a598\") "
Oct 30 18:43:06.127008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.126967    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca" pod "489389b0-0319-41b7-afcf-8ee8c92dbbec" (UID: "489389b0-0319-41b7-afcf-8ee8c92dbbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:06.127283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.127259    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.127559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.127471    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca" pod "89300a9e-0887-4e6b-8641-e093cd094c52" (UID: "89300a9e-0887-4e6b-8641-e093cd094c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:06.127688 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.127590    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca" pod "b0bc661f-c2c7-40cc-90bc-7eb69438a598" (UID: "b0bc661f-c2c7-40cc-90bc-7eb69438a598"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:06.130482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.130459    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca" pod "89300a9e-0887-4e6b-8641-e093cd094c52" (UID: "89300a9e-0887-4e6b-8641-e093cd094c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:06.130589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.130504    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d" pod "b0bc661f-c2c7-40cc-90bc-7eb69438a598" (UID: "b0bc661f-c2c7-40cc-90bc-7eb69438a598"). InnerVolumeSpecName "kube-api-access-zrg6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:06.130674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.130597    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx" pod "489389b0-0319-41b7-afcf-8ee8c92dbbec" (UID: "489389b0-0319-41b7-afcf-8ee8c92dbbec"). InnerVolumeSpecName "kube-api-access-5clrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:06.130859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.130831    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca" pod "489389b0-0319-41b7-afcf-8ee8c92dbbec" (UID: "489389b0-0319-41b7-afcf-8ee8c92dbbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:06.130924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.130873    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf" pod "89300a9e-0887-4e6b-8641-e093cd094c52" (UID: "89300a9e-0887-4e6b-8641-e093cd094c52"). InnerVolumeSpecName "kube-api-access-trktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:06.131069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.131048    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca" pod "b0bc661f-c2c7-40cc-90bc-7eb69438a598" (UID: "b0bc661f-c2c7-40cc-90bc-7eb69438a598"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:06.228894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228866    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trktf\" (UniqueName: \"kubernetes.io/projected/89300a9e-0887-4e6b-8641-e093cd094c52-kube-api-access-trktf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.228894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228893    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrg6d\" (UniqueName: \"kubernetes.io/projected/b0bc661f-c2c7-40cc-90bc-7eb69438a598-kube-api-access-zrg6d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228904    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489389b0-0319-41b7-afcf-8ee8c92dbbec-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228913    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228922    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228930    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5clrx\" (UniqueName: \"kubernetes.io/projected/489389b0-0319-41b7-afcf-8ee8c92dbbec-kube-api-access-5clrx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228939    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bc661f-c2c7-40cc-90bc-7eb69438a598-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.229099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.228949    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89300a9e-0887-4e6b-8641-e093cd094c52-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:06.607330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.607300    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:06.607739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.607714    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:06.608372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.608355    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:06.739495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.739166    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"
Oct 30 18:43:06.739495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.739161    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7" event={"ID":"89300a9e-0887-4e6b-8641-e093cd094c52","Type":"ContainerDied","Data":"03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266"}
Oct 30 18:43:06.739495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.739302    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03046d1b5cbf2add2a50d5207bf22ca33927ac24068ddeb71afa2bc221269266"
Oct 30 18:43:06.740808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.740790    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"
Oct 30 18:43:06.740937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.740828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b" event={"ID":"489389b0-0319-41b7-afcf-8ee8c92dbbec","Type":"ContainerDied","Data":"9c9d9cea89f3b38eb76346ec9a2d81bf9c392a222fb749631926a0e79cdc1e31"}
Oct 30 18:43:06.740937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.740872    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9d9cea89f3b38eb76346ec9a2d81bf9c392a222fb749631926a0e79cdc1e31"
Oct 30 18:43:06.746245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.746220    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"
Oct 30 18:43:06.746359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.746240    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw" event={"ID":"b0bc661f-c2c7-40cc-90bc-7eb69438a598","Type":"ContainerDied","Data":"a99bebb3150a210b82bed707fd3385e3f6c220cbebef38f7f5d80f685732a274"}
Oct 30 18:43:06.746359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:06.746280    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99bebb3150a210b82bed707fd3385e3f6c220cbebef38f7f5d80f685732a274"
Oct 30 18:43:07.764330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.764290    5065 generic.go:356] "Generic (PLEG): container finished" podID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerID="86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd" exitCode=0
Oct 30 18:43:07.764844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.764368    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerDied","Data":"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"}
Oct 30 18:43:07.764967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.764946    5065 scope.go:122] "RemoveContainer" containerID="86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"
Oct 30 18:43:07.766414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.766373    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:07.766526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.766505    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:07.766593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.766576    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:07.894229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894195    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"]
Oct 30 18:43:07.894938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894921    5065 cpu_manager.go:414] "containerMap: removing container" podUID="89300a9e-0887-4e6b-8641-e093cd094c52" containerName="revoke-keys"
Oct 30 18:43:07.894938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894939    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89300a9e-0887-4e6b-8641-e093cd094c52" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894957    5065 cpu_manager.go:414] "containerMap: removing container" podUID="489389b0-0319-41b7-afcf-8ee8c92dbbec" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894966    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489389b0-0319-41b7-afcf-8ee8c92dbbec" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894985    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0cabe4e2-1eec-47c0-9f8a-e94837963c25" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.894992    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cabe4e2-1eec-47c0-9f8a-e94837963c25" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895020    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b0bc661f-c2c7-40cc-90bc-7eb69438a598" containerName="revoke-keys"
Oct 30 18:43:07.895137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895027    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bc661f-c2c7-40cc-90bc-7eb69438a598" containerName="revoke-keys"
Oct 30 18:43:07.895448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895164    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="89300a9e-0887-4e6b-8641-e093cd094c52" containerName="revoke-keys"
Oct 30 18:43:07.895448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895178    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="489389b0-0319-41b7-afcf-8ee8c92dbbec" containerName="revoke-keys"
Oct 30 18:43:07.895448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895188    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cabe4e2-1eec-47c0-9f8a-e94837963c25" containerName="revoke-keys"
Oct 30 18:43:07.895448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.895201    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bc661f-c2c7-40cc-90bc-7eb69438a598" containerName="revoke-keys"
Oct 30 18:43:07.899361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.899319    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:07.906879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.906831    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:43:07.907658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.907639    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:43:07.909991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:07.909191    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"]
Oct 30 18:43:08.048924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.048843    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kvp\" (UniqueName: \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.048924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.048886    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.049124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.048967    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.149678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.149643    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.149828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.149799    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kvp\" (UniqueName: \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.149875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.149844    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.150424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.150404    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.152469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.152447    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.163410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.163384    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kvp\" (UniqueName: \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\") pod \"maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.220293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.220253    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:08.511807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.511777    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"]
Oct 30 18:43:08.512490 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:08.512466    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd34eec_4fa6_4c23_93fe_3a6e4f858e52.slice/crio-9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584 WatchSource:0}: Error finding container 9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584: Status 404 returned error can't find the container with id 9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584
Oct 30 18:43:08.772922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.772893    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerStarted","Data":"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"}
Oct 30 18:43:08.774152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.774122    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw" event={"ID":"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52","Type":"ContainerStarted","Data":"9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584"}
Oct 30 18:43:08.876755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.876674    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:08.876894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.876845    5065 patch_prober.go:29] interesting pod/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:43:08.876939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:08.876886    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 18:43:09.331874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:09.331840    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:43:09.803110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:09.793212    5065 generic.go:356] "Generic (PLEG): container finished" podID="7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" containerID="12d4fc5cc14bbd92f54ffa6ee93b4cf5e92f949c22459308fc88f8bd82df7f12" exitCode=0
Oct 30 18:43:09.803110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:09.794177    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw" event={"ID":"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52","Type":"ContainerDied","Data":"12d4fc5cc14bbd92f54ffa6ee93b4cf5e92f949c22459308fc88f8bd82df7f12"}
Oct 30 18:43:09.876663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:09.876581    5065 patch_prober.go:29] interesting pod/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:43:09.876663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:09.876634    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 18:43:10.057551 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:10.057510    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"]
Oct 30 18:43:10.060287 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:10.060246    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-78934afe-p468b"]
Oct 30 18:43:10.876621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:10.876584    5065 patch_prober.go:29] interesting pod/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:43:10.877110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:10.876644    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 18:43:11.067270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.067230    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:11.129651 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.129562    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:43:11.156231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.154126    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"]
Oct 30 18:43:11.163433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.163066    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7289525-pc6n7"]
Oct 30 18:43:11.219543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.219501    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\") pod \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") "
Oct 30 18:43:11.219736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.219655    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\") pod \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") "
Oct 30 18:43:11.219803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.219769    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\" (UniqueName: \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\") pod \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\" (UID: \"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52\") "
Oct 30 18:43:11.221403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.221372    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca" pod "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" (UID: "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:11.221807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.221784    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.222552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.222526    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca" pod "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" (UID: "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:11.224500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.224474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp" pod "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" (UID: "7dd34eec-4fa6-4c23-93fe-3a6e4f858e52"). InnerVolumeSpecName "kube-api-access-h2kvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:11.340331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.340254    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kvp\" (UniqueName: \"kubernetes.io/projected/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-kube-api-access-h2kvp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.340331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.340297    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.842249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.835403    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw" event={"ID":"7dd34eec-4fa6-4c23-93fe-3a6e4f858e52","Type":"ContainerDied","Data":"9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584"}
Oct 30 18:43:11.842249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.835518    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f730a3f374606f521577433fa766695ac2eb667c99c216bbbff6c6cdbfeb584"
Oct 30 18:43:11.842249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.835589    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"
Oct 30 18:43:11.877306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.877276    5065 patch_prober.go:29] interesting pod/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:43:11.877713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.877337    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 18:43:11.915243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.915211    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489389b0-0319-41b7-afcf-8ee8c92dbbec" path="/var/lib/kubelet/pods/489389b0-0319-41b7-afcf-8ee8c92dbbec/volumes"
Oct 30 18:43:11.915936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:11.915911    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89300a9e-0887-4e6b-8641-e093cd094c52" path="/var/lib/kubelet/pods/89300a9e-0887-4e6b-8641-e093cd094c52/volumes"
Oct 30 18:43:12.876544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:12.876506    5065 patch_prober.go:29] interesting pod/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:43:12.876692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:12.876563    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 18:43:12.940427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:12.940396    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:43:12.941167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:12.941139    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy" containerID="cri-o://aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5" gracePeriod=30
Oct 30 18:43:13.847440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:13.847406    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:43:13.856624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:13.856593    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:43:13.988207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:13.988176    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:13.988700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:13.988684    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:13.989692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:13.989665    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:14.332761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.332715    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" podUID="9de91640-3a67-4722-be4b-1d281c8b63f9" containerName="payload-processing" containerID="cri-o://e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49" gracePeriod=25
Oct 30 18:43:14.642040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.642006    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:43:14.678129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.678080    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\" (UniqueName: \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\") pod \"9de91640-3a67-4722-be4b-1d281c8b63f9\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") "
Oct 30 18:43:14.678252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.678204    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\") pod \"9de91640-3a67-4722-be4b-1d281c8b63f9\" (UID: \"9de91640-3a67-4722-be4b-1d281c8b63f9\") "
Oct 30 18:43:14.678583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.678547    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume" pod "9de91640-3a67-4722-be4b-1d281c8b63f9" (UID: "9de91640-3a67-4722-be4b-1d281c8b63f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:14.680802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.680763    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf" pod "9de91640-3a67-4722-be4b-1d281c8b63f9" (UID: "9de91640-3a67-4722-be4b-1d281c8b63f9"). InnerVolumeSpecName "kube-api-access-fq6nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:14.708491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.708468    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"]
Oct 30 18:43:14.711145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.711126    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9aded813-cr7nw"]
Oct 30 18:43:14.779049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.779020    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6nf\" (UniqueName: \"kubernetes.io/projected/9de91640-3a67-4722-be4b-1d281c8b63f9-kube-api-access-fq6nf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:14.779049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.779046    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de91640-3a67-4722-be4b-1d281c8b63f9-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:14.860905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.860874    5065 generic.go:356] "Generic (PLEG): container finished" podID="9de91640-3a67-4722-be4b-1d281c8b63f9" containerID="e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49" exitCode=0
Oct 30 18:43:14.861053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.860963    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" event={"ID":"9de91640-3a67-4722-be4b-1d281c8b63f9","Type":"ContainerDied","Data":"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"}
Oct 30 18:43:14.861053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.860992    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"
Oct 30 18:43:14.861189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.861063    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds" event={"ID":"9de91640-3a67-4722-be4b-1d281c8b63f9","Type":"ContainerDied","Data":"895390fcc34e96674072cfaf8bfcfc2112ca97ce76090eea1f68cdfe0c486dac"}
Oct 30 18:43:14.861189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.861126    5065 scope.go:122] "RemoveContainer" containerID="e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"
Oct 30 18:43:14.874144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.874125    5065 scope.go:122] "RemoveContainer" containerID="e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"
Oct 30 18:43:14.874412 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:14.874392    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49\": container with ID starting with e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49 not found: ID does not exist" containerID="e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"
Oct 30 18:43:14.874487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.874423    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49"} err="failed to get container status \"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49\": rpc error: code = NotFound desc = could not find container \"e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49\": container with ID starting with e944c2791b6d2a543011a9c13af7dfb59febebd2491317c84cb8ffcbdd512d49 not found: ID does not exist"
Oct 30 18:43:14.888902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.888881    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:43:14.896074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:14.896052    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7289525-6bc5bc4b7d-n99ds"]
Oct 30 18:43:15.019387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.019355    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"]
Oct 30 18:43:15.023680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.023657    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5768279-7jc8f"]
Oct 30 18:43:15.221797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.221713    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:43:15.221956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.221926    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" containerID="cri-o://8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55" gracePeriod=30
Oct 30 18:43:15.889867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.889835    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cabe4e2-1eec-47c0-9f8a-e94837963c25" path="/var/lib/kubelet/pods/0cabe4e2-1eec-47c0-9f8a-e94837963c25/volumes"
Oct 30 18:43:15.890372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.890352    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de91640-3a67-4722-be4b-1d281c8b63f9" path="/var/lib/kubelet/pods/9de91640-3a67-4722-be4b-1d281c8b63f9/volumes"
Oct 30 18:43:15.890861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:15.890845    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bc661f-c2c7-40cc-90bc-7eb69438a598" path="/var/lib/kubelet/pods/b0bc661f-c2c7-40cc-90bc-7eb69438a598/volumes"
Oct 30 18:43:16.130639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.130601    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" podUID="7a408941-edc5-4859-a647-1bc143c67952" containerName="payload-pre-processing" containerID="cri-o://b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63" gracePeriod=25
Oct 30 18:43:16.438201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.438173    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:43:16.493160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.493133    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\" (UniqueName: \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\") pod \"7a408941-edc5-4859-a647-1bc143c67952\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") "
Oct 30 18:43:16.493291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.493195    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\") pod \"7a408941-edc5-4859-a647-1bc143c67952\" (UID: \"7a408941-edc5-4859-a647-1bc143c67952\") "
Oct 30 18:43:16.493579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.493546    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume" pod "7a408941-edc5-4859-a647-1bc143c67952" (UID: "7a408941-edc5-4859-a647-1bc143c67952"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:16.493708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.493661    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a408941-edc5-4859-a647-1bc143c67952-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:16.495768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.495739    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6" pod "7a408941-edc5-4859-a647-1bc143c67952" (UID: "7a408941-edc5-4859-a647-1bc143c67952"). InnerVolumeSpecName "kube-api-access-ff7f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:16.594290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.594260    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7f6\" (UniqueName: \"kubernetes.io/projected/7a408941-edc5-4859-a647-1bc143c67952-kube-api-access-ff7f6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:16.763617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.763573    5065 patch_prober.go:29] interesting pod/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 18:43:16.763793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.763650    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:43:16.875810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.875777    5065 generic.go:356] "Generic (PLEG): container finished" podID="7a408941-edc5-4859-a647-1bc143c67952" containerID="b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63" exitCode=0
Oct 30 18:43:16.875978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.875825    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"
Oct 30 18:43:16.875978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.875858    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" event={"ID":"7a408941-edc5-4859-a647-1bc143c67952","Type":"ContainerDied","Data":"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"}
Oct 30 18:43:16.875978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.875914    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb" event={"ID":"7a408941-edc5-4859-a647-1bc143c67952","Type":"ContainerDied","Data":"e3443ae78571e836a83e7b758abe5a3ed142a02fe5628b0a7888fe9ce693f11d"}
Oct 30 18:43:16.875978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.875936    5065 scope.go:122] "RemoveContainer" containerID="b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"
Oct 30 18:43:16.889576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.889550    5065 scope.go:122] "RemoveContainer" containerID="b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"
Oct 30 18:43:16.890429 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:16.890372    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63\": container with ID starting with b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63 not found: ID does not exist" containerID="b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"
Oct 30 18:43:16.890540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.890440    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63"} err="failed to get container status \"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63\": rpc error: code = NotFound desc = could not find container \"b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63\": container with ID starting with b00b967d1282c700d38f40b31eebcecc76836021fe94246ebf9feb645a108f63 not found: ID does not exist"
Oct 30 18:43:16.915003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.914967    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:43:16.915873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915848    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7a408941-edc5-4859-a647-1bc143c67952" containerName="payload-pre-processing"
Oct 30 18:43:16.916021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915874    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a408941-edc5-4859-a647-1bc143c67952" containerName="payload-pre-processing"
Oct 30 18:43:16.916021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915913    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" containerName="revoke-keys"
Oct 30 18:43:16.916021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915922    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" containerName="revoke-keys"
Oct 30 18:43:16.916021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915959    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9de91640-3a67-4722-be4b-1d281c8b63f9" containerName="payload-processing"
Oct 30 18:43:16.916021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.915967    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de91640-3a67-4722-be4b-1d281c8b63f9" containerName="payload-processing"
Oct 30 18:43:16.916325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.916106    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de91640-3a67-4722-be4b-1d281c8b63f9" containerName="payload-processing"
Oct 30 18:43:16.916325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.916124    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a408941-edc5-4859-a647-1bc143c67952" containerName="payload-pre-processing"
Oct 30 18:43:16.916325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.916138    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" containerName="revoke-keys"
Oct 30 18:43:16.921364 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.921343    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:16.925275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.925254    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:43:16.934169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.934070    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-84c32a-openshift-default-dockercfg-k6dxc\""
Oct 30 18:43:16.939895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.939870    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:43:16.945046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:16.945028    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9aded813-8d6d55b87-wjkmb"]
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002646    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002727    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002755    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002788    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hhf\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002852    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002881    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002918    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002947    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.003273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.002985    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.103869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.103772    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.103881    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.103909    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.103935    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hhf\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.103976    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104004    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104048    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104114    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104168    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104464    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104508    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.104794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.104731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.105161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.105137    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.105278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.105258    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.106459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.106437    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.106999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.106960    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.119814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.119790    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hhf\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.122272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.122253    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.241957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.241916    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:17.551885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.551860    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:43:17.553203 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:17.553172    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0780df06_a621_4798_a156_306814b6ebf4.slice/crio-a278e8125a805da5c5c926a0e6f143afe66e30583ba0e00b80e180e1c070fdd7 WatchSource:0}: Error finding container a278e8125a805da5c5c926a0e6f143afe66e30583ba0e00b80e180e1c070fdd7: Status 404 returned error can't find the container with id a278e8125a805da5c5c926a0e6f143afe66e30583ba0e00b80e180e1c070fdd7
Oct 30 18:43:17.555560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.555530    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:17.555664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.555590    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:17.555664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.555632    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:17.890054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.889968    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a408941-edc5-4859-a647-1bc143c67952" path="/var/lib/kubelet/pods/7a408941-edc5-4859-a647-1bc143c67952/volumes"
Oct 30 18:43:17.890381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.890357    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerStarted","Data":"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"}
Oct 30 18:43:17.890479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.890417    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerStarted","Data":"a278e8125a805da5c5c926a0e6f143afe66e30583ba0e00b80e180e1c070fdd7"}
Oct 30 18:43:17.916061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:17.916002    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podStartSLOduration=1.91598387 podStartE2EDuration="1.91598387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:17.912374401 +0000 UTC m=+618.175839349" watchObservedRunningTime="2026-10-30 18:43:17.91598387 +0000 UTC m=+618.179448819"
Oct 30 18:43:18.033283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.033245    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:43:18.067130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.060157    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:43:18.250663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.250634    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:18.262634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.262605    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:18.263910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.263889    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:18.324643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324602    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324690    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324722    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324785    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324820    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324878    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:18.324968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324925    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.324968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.324961    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.325211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325030    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.325211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325061    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\") pod \"9b45917d-1145-449f-a26f-4fccb2e09057\" (UID: \"9b45917d-1145-449f-a26f-4fccb2e09057\") "
Oct 30 18:43:18.325211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325193    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:18.325356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325208    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:18.325689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325625    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.325689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325650    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.325689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325664    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.325914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.325719    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:18.326241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.326212    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:18.327246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.327225    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "kube-api-access-68nng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:18.327772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.327748    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:18.328285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.328260    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:18.328451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.328426    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo" pod "9b45917d-1145-449f-a26f-4fccb2e09057" (UID: "9b45917d-1145-449f-a26f-4fccb2e09057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.426977    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b45917d-1145-449f-a26f-4fccb2e09057-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.427003    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b45917d-1145-449f-a26f-4fccb2e09057-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.427012    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.427021    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nng\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-kube-api-access-68nng\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.427031    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b45917d-1145-449f-a26f-4fccb2e09057-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.427039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.427039    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b45917d-1145-449f-a26f-4fccb2e09057-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:18.848733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.848683    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" podUID="8bdb8d01-d52e-4791-b227-5ce437d298c7" containerName="payload-processing" containerID="cri-o://357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9" gracePeriod=25
Oct 30 18:43:18.857452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.857397    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" podUID="174b1bba-bcf1-4f82-b581-1602bd8b610b" containerName="payload-pre-processing" containerID="cri-o://90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51" gracePeriod=25
Oct 30 18:43:18.895152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895080    5065 generic.go:356] "Generic (PLEG): container finished" podID="9b45917d-1145-449f-a26f-4fccb2e09057" containerID="aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5" exitCode=0
Oct 30 18:43:18.895270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895173    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"
Oct 30 18:43:18.895270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895117    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerDied","Data":"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"}
Oct 30 18:43:18.895270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895254    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq" event={"ID":"9b45917d-1145-449f-a26f-4fccb2e09057","Type":"ContainerDied","Data":"0d06997447c51dbc0d73250b15c7103c521c2b123a7073667ebff1f3148c7387"}
Oct 30 18:43:18.895477 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895284    5065 scope.go:122] "RemoveContainer" containerID="aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"
Oct 30 18:43:18.895657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.895639    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:18.897924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.897904    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:18.969843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.969695    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:43:18.971773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.971636    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7289525-openshift-default-7b8c4cd545-rkmpq"]
Oct 30 18:43:18.977437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.977412    5065 scope.go:122] "RemoveContainer" containerID="540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"
Oct 30 18:43:18.989876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.989852    5065 scope.go:122] "RemoveContainer" containerID="aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"
Oct 30 18:43:18.995435 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:18.995399    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5\": container with ID starting with aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5 not found: ID does not exist" containerID="aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"
Oct 30 18:43:18.995571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.995454    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5"} err="failed to get container status \"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5\": rpc error: code = NotFound desc = could not find container \"aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5\": container with ID starting with aee493d5cd72d4088b3cbc8d2e09ca599d9eb810aa190bab2b2ec1ca08856bf5 not found: ID does not exist"
Oct 30 18:43:18.995571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.995487    5065 scope.go:122] "RemoveContainer" containerID="540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"
Oct 30 18:43:18.996309 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:18.996261    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4\": container with ID starting with 540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4 not found: ID does not exist" containerID="540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"
Oct 30 18:43:18.996309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:18.996289    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4"} err="failed to get container status \"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4\": rpc error: code = NotFound desc = could not find container \"540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4\": container with ID starting with 540038a1d48c7961cbbb94e7210cc2007da4208bde72c20ba6bf90bc8e617df4 not found: ID does not exist"
Oct 30 18:43:19.121049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.121020    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:43:19.179398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.179371    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:43:19.239543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.239512    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\" (UniqueName: \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\") pod \"8bdb8d01-d52e-4791-b227-5ce437d298c7\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") "
Oct 30 18:43:19.239692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.239665    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\") pod \"8bdb8d01-d52e-4791-b227-5ce437d298c7\" (UID: \"8bdb8d01-d52e-4791-b227-5ce437d298c7\") "
Oct 30 18:43:19.239731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.239705    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\") pod \"174b1bba-bcf1-4f82-b581-1602bd8b610b\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") "
Oct 30 18:43:19.239800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.239757    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\" (UniqueName: \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\") pod \"174b1bba-bcf1-4f82-b581-1602bd8b610b\" (UID: \"174b1bba-bcf1-4f82-b581-1602bd8b610b\") "
Oct 30 18:43:19.240218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.240172    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume" pod "8bdb8d01-d52e-4791-b227-5ce437d298c7" (UID: "8bdb8d01-d52e-4791-b227-5ce437d298c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:19.240218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.240205    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume" pod "174b1bba-bcf1-4f82-b581-1602bd8b610b" (UID: "174b1bba-bcf1-4f82-b581-1602bd8b610b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:19.240434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.240367    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdb8d01-d52e-4791-b227-5ce437d298c7-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:19.240434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.240390    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174b1bba-bcf1-4f82-b581-1602bd8b610b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:19.242048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.242026    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj" pod "8bdb8d01-d52e-4791-b227-5ce437d298c7" (UID: "8bdb8d01-d52e-4791-b227-5ce437d298c7"). InnerVolumeSpecName "kube-api-access-v2bkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:19.242436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.242416    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk" pod "174b1bba-bcf1-4f82-b581-1602bd8b610b" (UID: "174b1bba-bcf1-4f82-b581-1602bd8b610b"). InnerVolumeSpecName "kube-api-access-wlhvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:19.344158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.341210    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bkj\" (UniqueName: \"kubernetes.io/projected/8bdb8d01-d52e-4791-b227-5ce437d298c7-kube-api-access-v2bkj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:19.344158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.341246    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhvk\" (UniqueName: \"kubernetes.io/projected/174b1bba-bcf1-4f82-b581-1602bd8b610b-kube-api-access-wlhvk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:19.438561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.438488    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 18:43:19.439038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439023    5065 cpu_manager.go:414] "containerMap: removing container" podUID="174b1bba-bcf1-4f82-b581-1602bd8b610b" containerName="payload-pre-processing"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439040    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174b1bba-bcf1-4f82-b581-1602bd8b610b" containerName="payload-pre-processing"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439061    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439067    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439111    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439117    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439123    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8bdb8d01-d52e-4791-b227-5ce437d298c7" containerName="payload-processing"
Oct 30 18:43:19.439159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439128    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdb8d01-d52e-4791-b227-5ce437d298c7" containerName="payload-processing"
Oct 30 18:43:19.439386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439240    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.439386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439253    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdb8d01-d52e-4791-b227-5ce437d298c7" containerName="payload-processing"
Oct 30 18:43:19.439386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439269    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="174b1bba-bcf1-4f82-b581-1602bd8b610b" containerName="payload-pre-processing"
Oct 30 18:43:19.439386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.439283    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" containerName="istio-proxy"
Oct 30 18:43:19.442585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.442567    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.454366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.454341    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 18:43:19.454521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.454488    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-da70a5-dockercfg-d684v\""
Oct 30 18:43:19.455231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.455211    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-da70a5\""
Oct 30 18:43:19.478200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.478178    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 18:43:19.482744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.482724    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.497230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.495803    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 18:43:19.543788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.543758    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqtw\" (UniqueName: \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.543788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.543798    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.544015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.543971    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.544072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.544042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.644875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.644842    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.645120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.644951    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqtw\" (UniqueName: \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.645120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.644974    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.645120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.645031    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.645757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.645727    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.645850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.645732    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.659206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.659175    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqtw\" (UniqueName: \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\") pod \"payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.659509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.659486    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\") pod \"payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.759030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.759003    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:19.800961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.800927    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:19.894792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.894761    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b45917d-1145-449f-a26f-4fccb2e09057" path="/var/lib/kubelet/pods/9b45917d-1145-449f-a26f-4fccb2e09057/volumes"
Oct 30 18:43:19.915726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.915697    5065 generic.go:356] "Generic (PLEG): container finished" podID="174b1bba-bcf1-4f82-b581-1602bd8b610b" containerID="90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51" exitCode=0
Oct 30 18:43:19.915828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.915776    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" event={"ID":"174b1bba-bcf1-4f82-b581-1602bd8b610b","Type":"ContainerDied","Data":"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"}
Oct 30 18:43:19.915828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.915818    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl" event={"ID":"174b1bba-bcf1-4f82-b581-1602bd8b610b","Type":"ContainerDied","Data":"5ba69f108db959cc96aff57c76ba0b12edcb4266e5207cd517165011b3156060"}
Oct 30 18:43:19.916024 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.915869    5065 scope.go:122] "RemoveContainer" containerID="90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"
Oct 30 18:43:19.916075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.916059    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"
Oct 30 18:43:19.919660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.919628    5065 generic.go:356] "Generic (PLEG): container finished" podID="8bdb8d01-d52e-4791-b227-5ce437d298c7" containerID="357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9" exitCode=0
Oct 30 18:43:19.920447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.920427    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"
Oct 30 18:43:19.920824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.920802    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" event={"ID":"8bdb8d01-d52e-4791-b227-5ce437d298c7","Type":"ContainerDied","Data":"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"}
Oct 30 18:43:19.920909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.920848    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb" event={"ID":"8bdb8d01-d52e-4791-b227-5ce437d298c7","Type":"ContainerDied","Data":"c2be9b7a879e99539592313db9431f74e1a44ca15214b466397599a118d70005"}
Oct 30 18:43:19.941570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.941553    5065 scope.go:122] "RemoveContainer" containerID="90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"
Oct 30 18:43:19.941885 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:19.941860    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51\": container with ID starting with 90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51 not found: ID does not exist" containerID="90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"
Oct 30 18:43:19.941957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.941897    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51"} err="failed to get container status \"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51\": rpc error: code = NotFound desc = could not find container \"90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51\": container with ID starting with 90b33e935ead0ccce2bf62d20487eb796457b7abc16094e3b155811f23ed4f51 not found: ID does not exist"
Oct 30 18:43:19.941957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.941919    5065 scope.go:122] "RemoveContainer" containerID="357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"
Oct 30 18:43:19.957180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.957160    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:43:19.972872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.972293    5065 scope.go:122] "RemoveContainer" containerID="357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"
Oct 30 18:43:19.972872 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:19.972776    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9\": container with ID starting with 357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9 not found: ID does not exist" containerID="357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"
Oct 30 18:43:19.972872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.972812    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9"} err="failed to get container status \"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9\": rpc error: code = NotFound desc = could not find container \"357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9\": container with ID starting with 357796fe10005cb9e84c6d98562a94aeb774d92d0586ff0ee775729a45287dd9 not found: ID does not exist"
Oct 30 18:43:19.977278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.977216    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5768279-68467d757f-x2sbl"]
Oct 30 18:43:19.988655 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:19.988627    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:43:20.003579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.003546    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5768279-5bf8dd48fd-jkhfb"]
Oct 30 18:43:20.048566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.048538    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 18:43:20.048688 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:20.048650    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62818c2_bcc0_4b64_9af7_9be58e7a0895.slice/crio-1fd4ccc8bb9626dae67853ce2bce86877ceb95d3a9f7a86946fcfd195690d457 WatchSource:0}: Error finding container 1fd4ccc8bb9626dae67853ce2bce86877ceb95d3a9f7a86946fcfd195690d457: Status 404 returned error can't find the container with id 1fd4ccc8bb9626dae67853ce2bce86877ceb95d3a9f7a86946fcfd195690d457
Oct 30 18:43:20.114722 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:20.114691    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bc11f0_5045_4097_a098_c36c2fca7630.slice/crio-78a89c49bccc8c572292bc6b9fa5c1981d0d6f8927a52bbcfb42f41fe66491ae WatchSource:0}: Error finding container 78a89c49bccc8c572292bc6b9fa5c1981d0d6f8927a52bbcfb42f41fe66491ae: Status 404 returned error can't find the container with id 78a89c49bccc8c572292bc6b9fa5c1981d0d6f8927a52bbcfb42f41fe66491ae
Oct 30 18:43:20.115217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.115193    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 18:43:20.487443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.487417    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:43:20.559708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559675    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559716    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559740    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559780    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559807    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.559890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559857    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.560227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559922    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.560227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.559978    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\") pod \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\" (UID: \"378b82f2-e4bb-4afc-8e97-4d0ca8c34102\") "
Oct 30 18:43:20.560227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560019    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.560370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560224    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.560441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560421    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.560493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560439    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.560493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560447    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.560603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560510    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:20.560603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.560529    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.562499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.562462    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:20.562499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.562482    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.562499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.562488    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:43:20.562879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.562861    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc" pod "378b82f2-e4bb-4afc-8e97-4d0ca8c34102" (UID: "378b82f2-e4bb-4afc-8e97-4d0ca8c34102"). InnerVolumeSpecName "kube-api-access-89kzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:20.662300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662212    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662252    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662271    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89kzc\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-kube-api-access-89kzc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662287    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662302    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662317    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.662661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.662335    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378b82f2-e4bb-4afc-8e97-4d0ca8c34102-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.928495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.928410    5065 generic.go:356] "Generic (PLEG): container finished" podID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerID="8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55" exitCode=0
Oct 30 18:43:20.928495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.928475    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"
Oct 30 18:43:20.928703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.928492    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerDied","Data":"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"}
Oct 30 18:43:20.928703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.928565    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s" event={"ID":"378b82f2-e4bb-4afc-8e97-4d0ca8c34102","Type":"ContainerDied","Data":"8063572024cf862425f80ac03002a7daeab59c53e164226438a1665449bb561c"}
Oct 30 18:43:20.928703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.928593    5065 scope.go:122] "RemoveContainer" containerID="8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"
Oct 30 18:43:20.931032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.931005    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" event={"ID":"e62818c2-bcc0-4b64-9af7-9be58e7a0895","Type":"ContainerStarted","Data":"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"}
Oct 30 18:43:20.931317 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.931293    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" event={"ID":"e62818c2-bcc0-4b64-9af7-9be58e7a0895","Type":"ContainerStarted","Data":"1fd4ccc8bb9626dae67853ce2bce86877ceb95d3a9f7a86946fcfd195690d457"}
Oct 30 18:43:20.931440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.931338    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:20.933239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.933159    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" event={"ID":"08bc11f0-5045-4097-a098-c36c2fca7630","Type":"ContainerStarted","Data":"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"}
Oct 30 18:43:20.933239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.933201    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" event={"ID":"08bc11f0-5045-4097-a098-c36c2fca7630","Type":"ContainerStarted","Data":"78a89c49bccc8c572292bc6b9fa5c1981d0d6f8927a52bbcfb42f41fe66491ae"}
Oct 30 18:43:20.933239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.933231    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:20.942020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.942000    5065 scope.go:122] "RemoveContainer" containerID="2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"
Oct 30 18:43:20.952419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.952341    5065 scope.go:122] "RemoveContainer" containerID="8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"
Oct 30 18:43:20.952649 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:20.952627    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55\": container with ID starting with 8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55 not found: ID does not exist" containerID="8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"
Oct 30 18:43:20.952725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.952658    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55"} err="failed to get container status \"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55\": rpc error: code = NotFound desc = could not find container \"8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55\": container with ID starting with 8d87eeedfe75a05fae5eb7748f055372ecba245f3b43b9dc3d917ec718243d55 not found: ID does not exist"
Oct 30 18:43:20.952725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.952679    5065 scope.go:122] "RemoveContainer" containerID="2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"
Oct 30 18:43:20.952948 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:20.952933    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e\": container with ID starting with 2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e not found: ID does not exist" containerID="2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"
Oct 30 18:43:20.953000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.952950    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e"} err="failed to get container status \"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e\": rpc error: code = NotFound desc = could not find container \"2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e\": container with ID starting with 2f0d54f25a7b68484d0061ed5041cdf203115ac7e8657b54305790ca24e48a1e not found: ID does not exist"
Oct 30 18:43:20.972069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:20.972032    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" podStartSLOduration=1.972020674 podStartE2EDuration="1.972020674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:20.969963129 +0000 UTC m=+621.233428088" watchObservedRunningTime="2026-10-30 18:43:20.972020674 +0000 UTC m=+621.235485637"
Oct 30 18:43:21.001247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.001193    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" podStartSLOduration=2.001178033 podStartE2EDuration="2.001178033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:20.998443895 +0000 UTC m=+621.261908922" watchObservedRunningTime="2026-10-30 18:43:21.001178033 +0000 UTC m=+621.264642993"
Oct 30 18:43:21.038533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.038497    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:43:21.043188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.043164    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f5768279-openshift-default-6b774bb54f-6xl5s"]
Oct 30 18:43:21.775359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.775330    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 18:43:21.775826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.775813    5065 cpu_manager.go:414] "containerMap: removing container" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:21.775935 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.775827    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:21.775935 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.775935    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:21.776009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.775947    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:21.780322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.780303    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.791168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.791146    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 18:43:21.793948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.793927    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8a92cc\""
Oct 30 18:43:21.875558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.875529    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnwf\" (UniqueName: \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.875725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.875569    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.895944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.895903    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174b1bba-bcf1-4f82-b581-1602bd8b610b" path="/var/lib/kubelet/pods/174b1bba-bcf1-4f82-b581-1602bd8b610b/volumes"
Oct 30 18:43:21.896499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.896479    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" path="/var/lib/kubelet/pods/378b82f2-e4bb-4afc-8e97-4d0ca8c34102/volumes"
Oct 30 18:43:21.897148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.897126    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdb8d01-d52e-4791-b227-5ce437d298c7" path="/var/lib/kubelet/pods/8bdb8d01-d52e-4791-b227-5ce437d298c7/volumes"
Oct 30 18:43:21.976646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.976609    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnwf\" (UniqueName: \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.976646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.976650    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.977286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.977267    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:21.989953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:21.989929    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnwf\" (UniqueName: \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\") pod \"payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:22.096582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.096503    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:22.384913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.384880    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 18:43:22.385034 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:22.384947    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a6702b_966a_4813_875e_c148ffbdc62f.slice/crio-ce1f8b805414a50b62aff48a3673248fd69cc0635e94c4d392eb83eb541c5eff WatchSource:0}: Error finding container ce1f8b805414a50b62aff48a3673248fd69cc0635e94c4d392eb83eb541c5eff: Status 404 returned error can't find the container with id ce1f8b805414a50b62aff48a3673248fd69cc0635e94c4d392eb83eb541c5eff
Oct 30 18:43:22.472441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.472412    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 18:43:22.472920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.472907    5065 cpu_manager.go:414] "containerMap: removing container" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:22.472958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.472921    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378b82f2-e4bb-4afc-8e97-4d0ca8c34102" containerName="istio-proxy"
Oct 30 18:43:22.476196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.476181    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.495756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.495728    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 18:43:22.539620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.539567    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-84c32a\""
Oct 30 18:43:22.562240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.562071    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-84c32a-dockercfg-wcktw\""
Oct 30 18:43:22.582981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.582942    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.583389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.583015    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklw8\" (UniqueName: \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.684047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.683965    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.684047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.684034    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklw8\" (UniqueName: \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.684932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.684904    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.700214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.700183    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklw8\" (UniqueName: \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\") pod \"payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.784528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.784501    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"]
Oct 30 18:43:22.787907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.787879    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72bc98fe-86ltw"]
Oct 30 18:43:22.793453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.793432    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:22.955049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.954944    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" event={"ID":"35a6702b-966a-4813-875e-c148ffbdc62f","Type":"ContainerStarted","Data":"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"}
Oct 30 18:43:22.955337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.955305    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" event={"ID":"35a6702b-966a-4813-875e-c148ffbdc62f","Type":"ContainerStarted","Data":"ce1f8b805414a50b62aff48a3673248fd69cc0635e94c4d392eb83eb541c5eff"}
Oct 30 18:43:22.955471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:22.955371    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:23.026448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.026367    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" podStartSLOduration=2.026344086 podStartE2EDuration="2.026344086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:22.996735912 +0000 UTC m=+623.260200894" watchObservedRunningTime="2026-10-30 18:43:23.026344086 +0000 UTC m=+623.289809035"
Oct 30 18:43:23.035391 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.035355    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" podUID="ad90645b-6109-4282-a3ad-d33657aa5705" containerName="payload-processing" containerID="cri-o://656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d" gracePeriod=25
Oct 30 18:43:23.037502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.037465    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:43:23.037997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.037970    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy" containerID="cri-o://574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5" gracePeriod=30
Oct 30 18:43:23.069386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.069348    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" podUID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" containerName="payload-pre-processing" containerID="cri-o://600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa" gracePeriod=25
Oct 30 18:43:23.108689 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:23.108312    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:43:23.145135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.138110    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 18:43:23.429194 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.429155    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:43:23.479686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.479618    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:43:23.521950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.521921    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\" (UniqueName: \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\") pod \"ad90645b-6109-4282-a3ad-d33657aa5705\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") "
Oct 30 18:43:23.522181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.521963    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\") pod \"ad90645b-6109-4282-a3ad-d33657aa5705\" (UID: \"ad90645b-6109-4282-a3ad-d33657aa5705\") "
Oct 30 18:43:23.523382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.523342    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume" pod "ad90645b-6109-4282-a3ad-d33657aa5705" (UID: "ad90645b-6109-4282-a3ad-d33657aa5705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:23.525366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.525340    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98" pod "ad90645b-6109-4282-a3ad-d33657aa5705" (UID: "ad90645b-6109-4282-a3ad-d33657aa5705"). InnerVolumeSpecName "kube-api-access-6sz98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:23.623372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.623336    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\" (UniqueName: \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\") pod \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") "
Oct 30 18:43:23.623560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.623457    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\") pod \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\" (UID: \"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5\") "
Oct 30 18:43:23.623785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.623765    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sz98\" (UniqueName: \"kubernetes.io/projected/ad90645b-6109-4282-a3ad-d33657aa5705-kube-api-access-6sz98\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:23.623838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.623791    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad90645b-6109-4282-a3ad-d33657aa5705-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:23.623887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.623858    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume" pod "dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" (UID: "dc413de6-1d91-4b24-a1e2-4d3afbdd73a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:23.625622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.625601    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5" pod "dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" (UID: "dc413de6-1d91-4b24-a1e2-4d3afbdd73a5"). InnerVolumeSpecName "kube-api-access-frjx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:23.724811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.724783    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjx5\" (UniqueName: \"kubernetes.io/projected/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-kube-api-access-frjx5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:23.724811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.724811    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:23.890185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.890151    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd34eec-4fa6-4c23-93fe-3a6e4f858e52" path="/var/lib/kubelet/pods/7dd34eec-4fa6-4c23-93fe-3a6e4f858e52/volumes"
Oct 30 18:43:23.962813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.962783    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" event={"ID":"8e7a7995-a027-4057-8b41-4882ab30c912","Type":"ContainerStarted","Data":"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"}
Oct 30 18:43:23.962934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.962853    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:23.962934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.962883    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" event={"ID":"8e7a7995-a027-4057-8b41-4882ab30c912","Type":"ContainerStarted","Data":"b9cd56373b9d141b75dd4e6be1109bfc1d04a966c22c4a40ef8c6cf3909e3533"}
Oct 30 18:43:23.964131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.964105    5065 generic.go:356] "Generic (PLEG): container finished" podID="ad90645b-6109-4282-a3ad-d33657aa5705" containerID="656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d" exitCode=0
Oct 30 18:43:23.964209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.964138    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" event={"ID":"ad90645b-6109-4282-a3ad-d33657aa5705","Type":"ContainerDied","Data":"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"}
Oct 30 18:43:23.964209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.964154    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"
Oct 30 18:43:23.964209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.964185    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7" event={"ID":"ad90645b-6109-4282-a3ad-d33657aa5705","Type":"ContainerDied","Data":"4fd9276196fff905e1398a358bbf21bc857b5891e3d098648288d2d3957a9951"}
Oct 30 18:43:23.964323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.964214    5065 scope.go:122] "RemoveContainer" containerID="656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"
Oct 30 18:43:23.966223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.966200    5065 generic.go:356] "Generic (PLEG): container finished" podID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" containerID="600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa" exitCode=0
Oct 30 18:43:23.966345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.966263    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"
Oct 30 18:43:23.966345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.966302    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" event={"ID":"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5","Type":"ContainerDied","Data":"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"}
Oct 30 18:43:23.966517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.966352    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz" event={"ID":"dc413de6-1d91-4b24-a1e2-4d3afbdd73a5","Type":"ContainerDied","Data":"e08b930938c9a4cb6eebe5f3a952590a3cb4558550e7efc6ea6a4157111da4e7"}
Oct 30 18:43:23.976413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.976388    5065 scope.go:122] "RemoveContainer" containerID="656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"
Oct 30 18:43:23.976718 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:23.976699    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d\": container with ID starting with 656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d not found: ID does not exist" containerID="656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"
Oct 30 18:43:23.976787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.976724    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d"} err="failed to get container status \"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d\": rpc error: code = NotFound desc = could not find container \"656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d\": container with ID starting with 656ae4f17e613356399206175603449d641e478af29c3dce279621bcbcd5585d not found: ID does not exist"
Oct 30 18:43:23.976787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.976742    5065 scope.go:122] "RemoveContainer" containerID="600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"
Oct 30 18:43:23.986274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.986257    5065 scope.go:122] "RemoveContainer" containerID="600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"
Oct 30 18:43:23.986555 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:23.986537    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa\": container with ID starting with 600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa not found: ID does not exist" containerID="600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"
Oct 30 18:43:23.986602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.986562    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa"} err="failed to get container status \"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa\": rpc error: code = NotFound desc = could not find container \"600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa\": container with ID starting with 600e6a9a2646aeae2b2484cb6dd1d9d312e76696a350ae17922984770ee87faa not found: ID does not exist"
Oct 30 18:43:23.990922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:23.990886    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" podStartSLOduration=1.990874969 podStartE2EDuration="1.990874969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:23.988024008 +0000 UTC m=+624.251488957" watchObservedRunningTime="2026-10-30 18:43:23.990874969 +0000 UTC m=+624.254339917"
Oct 30 18:43:24.006930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.006902    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:43:24.013429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.013407    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72bc98fe-7c56b86bcb-2cvf7"]
Oct 30 18:43:24.030037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.030019    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:43:24.032380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.032361    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72bc98fe-7b78f5899f-pqjjz"]
Oct 30 18:43:24.399132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399100    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 18:43:24.399705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399685    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ad90645b-6109-4282-a3ad-d33657aa5705" containerName="payload-processing"
Oct 30 18:43:24.399705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399702    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad90645b-6109-4282-a3ad-d33657aa5705" containerName="payload-processing"
Oct 30 18:43:24.399913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399737    5065 cpu_manager.go:414] "containerMap: removing container" podUID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" containerName="payload-pre-processing"
Oct 30 18:43:24.399913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399747    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" containerName="payload-pre-processing"
Oct 30 18:43:24.399913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399829    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" containerName="payload-pre-processing"
Oct 30 18:43:24.399913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.399842    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad90645b-6109-4282-a3ad-d33657aa5705" containerName="payload-processing"
Oct 30 18:43:24.404522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.404504    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.432877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.432851    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 18:43:24.433146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.433131    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-aa6736-openshift-default-dockercfg-9lfh6\""
Oct 30 18:43:24.532333 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532302    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532355    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532438    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532482    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532503    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532604    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532656    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532708    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8hb9\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.532790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.532772    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634260    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634321    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634338    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8hb9\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634374    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634400    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634432    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634470    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634512    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634541    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634760    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.634897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.634769    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.635209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.635184    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.635331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.635185    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.635426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.635338    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.637187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.637169    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.637346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.637311    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.648244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.648219    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8hb9\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.648358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.648263    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:24.718385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:24.718303    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:25.013133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.013080    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 18:43:25.017386 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:25.017363    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7956b75_cb3b_4443_9c51_c3c8d9dd2167.slice/crio-4642ec85d3712dd3833e6ee93dd05cfd4df04f6563c8bc693e6e0b9be396e81a WatchSource:0}: Error finding container 4642ec85d3712dd3833e6ee93dd05cfd4df04f6563c8bc693e6e0b9be396e81a: Status 404 returned error can't find the container with id 4642ec85d3712dd3833e6ee93dd05cfd4df04f6563c8bc693e6e0b9be396e81a
Oct 30 18:43:25.019700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.019664    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:25.019802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.019735    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:25.019802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.019765    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:25.890824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.890792    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad90645b-6109-4282-a3ad-d33657aa5705" path="/var/lib/kubelet/pods/ad90645b-6109-4282-a3ad-d33657aa5705/volumes"
Oct 30 18:43:25.891356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.891339    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc413de6-1d91-4b24-a1e2-4d3afbdd73a5" path="/var/lib/kubelet/pods/dc413de6-1d91-4b24-a1e2-4d3afbdd73a5/volumes"
Oct 30 18:43:25.990958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.990920    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerStarted","Data":"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"}
Oct 30 18:43:25.991156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:25.990984    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerStarted","Data":"4642ec85d3712dd3833e6ee93dd05cfd4df04f6563c8bc693e6e0b9be396e81a"}
Oct 30 18:43:26.021099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:26.021029    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podStartSLOduration=2.021014386 podStartE2EDuration="2.021014386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:26.018209741 +0000 UTC m=+626.281674690" watchObservedRunningTime="2026-10-30 18:43:26.021014386 +0000 UTC m=+626.284479331"
Oct 30 18:43:26.718904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:26.718859    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:26.724872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:26.724847    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:26.998759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:26.998684    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:26.999877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:26.999859    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:28.252485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.252456    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:28.376786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376716    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.376786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376775    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376837    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376873    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.376964    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377014    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377054    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377049    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:28.377239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377113    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\") pod \"f4842e58-a7a7-415e-b12c-08e9c536915e\" (UID: \"f4842e58-a7a7-415e-b12c-08e9c536915e\") "
Oct 30 18:43:28.377239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377198    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:28.377502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377298    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:28.377502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377456    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:28.377604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377566    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.377604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377579    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:28.377604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377585    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.377970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377622    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4842e58-a7a7-415e-b12c-08e9c536915e-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.377970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.377639    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.380875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.379928    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:43:28.381748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.380209    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:28.382350 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.382322    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:28.382656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.382629    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9" pod "f4842e58-a7a7-415e-b12c-08e9c536915e" (UID: "f4842e58-a7a7-415e-b12c-08e9c536915e"). InnerVolumeSpecName "kube-api-access-n5dd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:28.479063 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.479035    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.479063 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.479062    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dd9\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-kube-api-access-n5dd9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.479319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.479075    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.479319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.479107    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:28.479319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:28.479120    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4842e58-a7a7-415e-b12c-08e9c536915e-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:29.015276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.015236    5065 generic.go:356] "Generic (PLEG): container finished" podID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerID="574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5" exitCode=0
Oct 30 18:43:29.015444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.015300    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"
Oct 30 18:43:29.015444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.015317    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerDied","Data":"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"}
Oct 30 18:43:29.015444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.015377    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n" event={"ID":"f4842e58-a7a7-415e-b12c-08e9c536915e","Type":"ContainerDied","Data":"17f2ef5fde0aec1930b09b8a5387b81fc10af00415bb7514de12b2e717861738"}
Oct 30 18:43:29.015444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.015405    5065 scope.go:122] "RemoveContainer" containerID="574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"
Oct 30 18:43:29.034061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.034040    5065 scope.go:122] "RemoveContainer" containerID="86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"
Oct 30 18:43:29.043471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.043455    5065 scope.go:122] "RemoveContainer" containerID="574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"
Oct 30 18:43:29.043756 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:29.043738    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5\": container with ID starting with 574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5 not found: ID does not exist" containerID="574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"
Oct 30 18:43:29.043800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.043765    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5"} err="failed to get container status \"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5\": rpc error: code = NotFound desc = could not find container \"574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5\": container with ID starting with 574a7e6aaa014e73b26fdf11a512ce51f387c85070f1b662718325a3983fbfc5 not found: ID does not exist"
Oct 30 18:43:29.043800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.043784    5065 scope.go:122] "RemoveContainer" containerID="86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"
Oct 30 18:43:29.044026 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:29.044006    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd\": container with ID starting with 86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd not found: ID does not exist" containerID="86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"
Oct 30 18:43:29.044071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.044035    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd"} err="failed to get container status \"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd\": rpc error: code = NotFound desc = could not find container \"86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd\": container with ID starting with 86c5fbf5cfb3bddeff0753b101f461fa3ced9b8e1f9475ccea26b50099ba4dfd not found: ID does not exist"
Oct 30 18:43:29.065367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.065343    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:43:29.076549 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.076526    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-72bc98fe-openshift-default-86c4bb6785-6ck4n"]
Oct 30 18:43:29.828384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.828345    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 18:43:29.828912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.828898    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.828955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.828913    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.828955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.828943    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.828955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.828949    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.829047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.829035    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.834075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.834055    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:29.848605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.848586    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 18:43:29.848973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.848948    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-aa6736-dockercfg-rmm2d\""
Oct 30 18:43:29.849120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.849012    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-aa6736\""
Oct 30 18:43:29.849335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.849316    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" containerName="istio-proxy"
Oct 30 18:43:29.852518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.852495    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 18:43:29.852613 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.852602    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:29.868138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.868114    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 18:43:29.928113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:29.923403    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4842e58-a7a7-415e-b12c-08e9c536915e" path="/var/lib/kubelet/pods/f4842e58-a7a7-415e-b12c-08e9c536915e/volumes"
Oct 30 18:43:30.013781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.013743    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbjs\" (UniqueName: \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.013971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.013792    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.013971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.013844    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.013971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.013872    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcbm\" (UniqueName: \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.115416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.115308    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbjs\" (UniqueName: \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.115416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.115372    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.115706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.115448    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.115706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.115487    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcbm\" (UniqueName: \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.116763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.116735    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.116882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.116824    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.129796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.129768    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcbm\" (UniqueName: \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\") pod \"payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.129990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.129968    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbjs\" (UniqueName: \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\") pod \"payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.150243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.150223    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:30.166228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.166080    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:30.468296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.468272    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 18:43:30.469422 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:30.469389    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565da0d2_ac21_468f_aee9_af8467737307.slice/crio-9e5fc20bc6420b0a66a11282cb67adc372c752b9084e4c752dbc5b8f318d7f92 WatchSource:0}: Error finding container 9e5fc20bc6420b0a66a11282cb67adc372c752b9084e4c752dbc5b8f318d7f92: Status 404 returned error can't find the container with id 9e5fc20bc6420b0a66a11282cb67adc372c752b9084e4c752dbc5b8f318d7f92
Oct 30 18:43:30.535674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:30.535645    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 18:43:30.537869 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:30.537838    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bee4cb_9318_42b1_b336_f664037b57e2.slice/crio-a457a66867576ea0b9298ee7bec5e9ef1c72279b7ecece8f21f1051f59ef732a WatchSource:0}: Error finding container a457a66867576ea0b9298ee7bec5e9ef1c72279b7ecece8f21f1051f59ef732a: Status 404 returned error can't find the container with id a457a66867576ea0b9298ee7bec5e9ef1c72279b7ecece8f21f1051f59ef732a
Oct 30 18:43:31.036556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.036522    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" event={"ID":"565da0d2-ac21-468f-aee9-af8467737307","Type":"ContainerStarted","Data":"7b507b7880e76f7c400445d8683e20c9511e038865272893cf70fe9c3ffd1a80"}
Oct 30 18:43:31.037049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.036592    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:31.037049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.036622    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" event={"ID":"565da0d2-ac21-468f-aee9-af8467737307","Type":"ContainerStarted","Data":"9e5fc20bc6420b0a66a11282cb67adc372c752b9084e4c752dbc5b8f318d7f92"}
Oct 30 18:43:31.038480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.038454    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" event={"ID":"94bee4cb-9318-42b1-b336-f664037b57e2","Type":"ContainerStarted","Data":"d82c7f9f774ed0f309e4b74b8f06e0a6f0a630203e2f556ab9cfe3b0c3e17f99"}
Oct 30 18:43:31.038589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.038504    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" event={"ID":"94bee4cb-9318-42b1-b336-f664037b57e2","Type":"ContainerStarted","Data":"a457a66867576ea0b9298ee7bec5e9ef1c72279b7ecece8f21f1051f59ef732a"}
Oct 30 18:43:31.038589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.038533    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:31.066328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.066289    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" podStartSLOduration=2.066276721 podStartE2EDuration="2.066276721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:31.063653977 +0000 UTC m=+631.327118925" watchObservedRunningTime="2026-10-30 18:43:31.066276721 +0000 UTC m=+631.329741708"
Oct 30 18:43:31.104714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.104655    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" podStartSLOduration=2.104635951 podStartE2EDuration="2.104635951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:31.099301366 +0000 UTC m=+631.362766315" watchObservedRunningTime="2026-10-30 18:43:31.104635951 +0000 UTC m=+631.368100901"
Oct 30 18:43:31.949042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.949016    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 18:43:31.949604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:31.949586    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 18:43:33.970284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:33.970257    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 18:43:35.036652 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:35.036624    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 18:43:39.101454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.101370    5065 generic.go:356] "Generic (PLEG): container finished" podID="0780df06-a621-4798-a156-306814b6ebf4" containerID="5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d" exitCode=0
Oct 30 18:43:39.101883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.101448    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerDied","Data":"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"}
Oct 30 18:43:39.101964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.101945    5065 scope.go:122] "RemoveContainer" containerID="5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"
Oct 30 18:43:39.103730 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.103699    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:39.103822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.103790    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:39.103864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:39.103840    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:40.130501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.130468    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerStarted","Data":"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"}
Oct 30 18:43:40.170736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.170706    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 18:43:40.174911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.174888    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.235111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.231205    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"kube-root-ca.crt\""
Oct 30 18:43:40.235111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.231399    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"simulator-w4-da70a5-kserve-self-signed-certs\""
Oct 30 18:43:40.235111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.231501    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"default-dockercfg-v4s5d\""
Oct 30 18:43:40.235111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.233204    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"openshift-service-ca.crt\""
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.248939    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.249007    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.249104    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.249156    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.249881    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4sp\" (UniqueName: \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.249951    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.250017    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.250042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.250066    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.262403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.250119    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.351862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351826    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4sp\" (UniqueName: \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351873    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351904    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351924    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351939    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.351960    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.352327    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.352677    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.352976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.352954    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.355279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.355252    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.355384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.355316    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.370270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.369133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4sp\" (UniqueName: \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\") pod \"simulator-w4-da70a5-kserve-75dc4d8589-4h4dq\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") " pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:40.575514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:40.575475    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:41.061371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:41.061341    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 18:43:41.152669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:41.152191    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" event={"ID":"e3e44f70-dd7c-412a-be3d-6a2260040484","Type":"ContainerStarted","Data":"53eddc2dc6c5b5b910831d70f2ad737fb403ef03a6f24e7ec4ceefc87ceeb154"}
Oct 30 18:43:41.249320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:41.247703    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:41.249320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:41.247760    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:42.056108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.054135    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 18:43:42.064264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.064243    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 18:43:42.174711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.174620    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" event={"ID":"e3e44f70-dd7c-412a-be3d-6a2260040484","Type":"ContainerStarted","Data":"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"}
Oct 30 18:43:42.249308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.248662    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:42.249308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.248759    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:42.335625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.335478    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podStartSLOduration=2.335453748 podStartE2EDuration="2.335453748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:42.318236148 +0000 UTC m=+642.581701097" watchObservedRunningTime="2026-10-30 18:43:42.335453748 +0000 UTC m=+642.598918700"
Oct 30 18:43:42.575879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:42.575839    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:43.242457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:43.242419    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:43.242933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:43.242498    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:44.244112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:44.242533    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:44.244112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:44.242591    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:45.243413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:45.243013    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:43:45.243413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:45.243105    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 18:43:46.247513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:46.247424    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:46.247975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:46.247959    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:46.248634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:46.248612    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:43:49.262227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.262196    5065 generic.go:356] "Generic (PLEG): container finished" podID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerID="c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f" exitCode=0
Oct 30 18:43:49.262587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.262279    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerDied","Data":"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"}
Oct 30 18:43:49.262745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.262728    5065 scope.go:122] "RemoveContainer" containerID="c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"
Oct 30 18:43:49.264126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.264101    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:49.264200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.264167    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:49.264200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:49.264197    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:43:50.272887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.272843    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerStarted","Data":"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"}
Oct 30 18:43:50.575854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.575777    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:50.614196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.614168    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:50.719491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.719462    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:50.719783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.719758    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:43:50.719843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:50.719823    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 18:43:51.315812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:51.315772    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:43:51.719272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:51.719182    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:43:51.719445 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:51.719275    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 18:43:52.612203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.612174    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 18:43:52.618799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.618761    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 18:43:52.618954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.618874    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.631239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.631216    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"kube-root-ca.crt\""
Oct 30 18:43:52.632275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.632249    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"default-dockercfg-7ql74\""
Oct 30 18:43:52.632390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.632284    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"openshift-service-ca.crt\""
Oct 30 18:43:52.632390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.632334    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"simulator-w0-84c32a-kserve-self-signed-certs\""
Oct 30 18:43:52.704136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.703611    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.704136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.703760    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz87m\" (UniqueName: \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.704136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.703842    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.704136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.704030    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.704487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.704076    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.704487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.704229    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.719081 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.719049    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:43:52.719212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.719128    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 18:43:52.805774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.805730    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.806553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.806531    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.807241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.807221    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.807788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.807767    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.808299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.808280    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz87m\" (UniqueName: \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.808491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.808471    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.820307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.808209    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.820503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.807672    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.820615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.806464    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.826561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.823527    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.827786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.826759    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.827786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.827173    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz87m\" (UniqueName: \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\") pod \"simulator-w0-84c32a-kserve-765556f5c8-95w7f\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") " pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:52.936813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:52.936733    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:53.070574 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:43:53.070535    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:43:53.250032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:53.250000    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 18:43:53.250215 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:43:53.250080    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4245d03_203c_4837_96d0_69e96e66b522.slice/crio-7b4ecfa432306b5031478300f06d120d2c5f24e04b2028acc91284416f344d03 WatchSource:0}: Error finding container 7b4ecfa432306b5031478300f06d120d2c5f24e04b2028acc91284416f344d03: Status 404 returned error can't find the container with id 7b4ecfa432306b5031478300f06d120d2c5f24e04b2028acc91284416f344d03
Oct 30 18:43:53.318890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:53.318860    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" event={"ID":"b4245d03-203c-4837-96d0-69e96e66b522","Type":"ContainerStarted","Data":"7b4ecfa432306b5031478300f06d120d2c5f24e04b2028acc91284416f344d03"}
Oct 30 18:43:53.720010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:53.719977    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:43:53.720432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:53.720029    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 18:43:54.328270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:54.328229    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" event={"ID":"b4245d03-203c-4837-96d0-69e96e66b522","Type":"ContainerStarted","Data":"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"}
Oct 30 18:43:54.352220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:54.352172    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podStartSLOduration=2.352157932 podStartE2EDuration="2.352157932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:54.350802054 +0000 UTC m=+654.614267022" watchObservedRunningTime="2026-10-30 18:43:54.352157932 +0000 UTC m=+654.615622889"
Oct 30 18:43:54.719731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:54.719656    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 18:43:54.720000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:54.719973    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 18:43:54.937321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:54.937286    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:43:55.725240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:55.725184    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:55.726806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:55.726786    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:43:55.730367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:43:55.730345    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 18:44:02.937262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:02.937228    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:44:02.954921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:02.954899    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:44:03.414584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.414546    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:44:03.774582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.774549    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 18:44:03.781890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.781861    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.784255 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.784228    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 18:44:03.794709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.794661    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"premium-w4-da70a5-kserve-self-signed-certs\""
Oct 30 18:44:03.838813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.838792    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.838930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.838831    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.838930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.838858    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmp58\" (UniqueName: \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.838930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.838911    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.839142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.839077    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.839208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.839162    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940288    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940339    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940404    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940447    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940497    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmp58\" (UniqueName: \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940518    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.940764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.941035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940768    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.941035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.940839    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.943191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.943162    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.943303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.943206    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:03.954931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:03.954911    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmp58\" (UniqueName: \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\") pod \"premium-w4-da70a5-kserve-5959dbc688-hc2d9\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") " pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:04.099967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:04.099840    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:04.405240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:04.405210    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 18:44:04.407326 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:04.407297    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod556e9d5c_0c3c_4ae8_8c7f_435d9a95120a.slice/crio-1daab8f6222165f9f97787d4b602207d732f98b22a0ae943fc82ae61d381890c WatchSource:0}: Error finding container 1daab8f6222165f9f97787d4b602207d732f98b22a0ae943fc82ae61d381890c: Status 404 returned error can't find the container with id 1daab8f6222165f9f97787d4b602207d732f98b22a0ae943fc82ae61d381890c
Oct 30 18:44:05.414406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:05.414363    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" event={"ID":"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a","Type":"ContainerStarted","Data":"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"}
Oct 30 18:44:05.414820 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:05.414416    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" event={"ID":"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a","Type":"ContainerStarted","Data":"1daab8f6222165f9f97787d4b602207d732f98b22a0ae943fc82ae61d381890c"}
Oct 30 18:44:05.439578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:05.439533    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podStartSLOduration=2.439520609 podStartE2EDuration="2.439520609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:05.435694245 +0000 UTC m=+665.699159220" watchObservedRunningTime="2026-10-30 18:44:05.439520609 +0000 UTC m=+665.702985557"
Oct 30 18:44:06.100278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:06.100249    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:14.100487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:14.100402    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:14.112773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:14.112748    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:14.483468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:14.483386    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:44:22.467482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.467444    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 18:44:22.471919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.471896    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.482708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.482687    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"premium-w0-84c32a-kserve-self-signed-certs\""
Oct 30 18:44:22.496470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.496440    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 18:44:22.631700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.631658    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.632003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.631976    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.632151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.632031    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.632215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.632171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.632271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.632212    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6hc\" (UniqueName: \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.632424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.632403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733173    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733209    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733234    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733268    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733319    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733342    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6hc\" (UniqueName: \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.733645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733630    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.734048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.733957    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.734048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.734067    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.736373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.736351    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.736711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.736688    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.749747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.749720    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6hc\" (UniqueName: \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\") pod \"premium-w0-84c32a-kserve-bcccf7bc9-wzc89\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") " pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:22.808245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:22.808222    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:23.065541 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:44:23.065506    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:44:23.089991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:23.089971    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 18:44:23.090969 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:23.090940    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfed93ae_e928_4a4d_9bb2_00a25f1989b6.slice/crio-52a5ad3459076e8f15f796e43b7521df5f2f2a2b94c739164e95f199d00d2988 WatchSource:0}: Error finding container 52a5ad3459076e8f15f796e43b7521df5f2f2a2b94c739164e95f199d00d2988: Status 404 returned error can't find the container with id 52a5ad3459076e8f15f796e43b7521df5f2f2a2b94c739164e95f199d00d2988
Oct 30 18:44:23.544332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:23.544305    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" event={"ID":"cfed93ae-e928-4a4d-9bb2-00a25f1989b6","Type":"ContainerStarted","Data":"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"}
Oct 30 18:44:23.544707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:23.544349    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" event={"ID":"cfed93ae-e928-4a4d-9bb2-00a25f1989b6","Type":"ContainerStarted","Data":"52a5ad3459076e8f15f796e43b7521df5f2f2a2b94c739164e95f199d00d2988"}
Oct 30 18:44:23.581261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:23.581212    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podStartSLOduration=1.5811994710000001 podStartE2EDuration="1.581199471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:23.57838085 +0000 UTC m=+683.841845815" watchObservedRunningTime="2026-10-30 18:44:23.581199471 +0000 UTC m=+683.844664460"
Oct 30 18:44:23.808722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:23.808652    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:26.835305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.835271    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 18:44:26.839446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.839428    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.846040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.846020    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7a8fa2-openshift-default-dockercfg-dddfr\""
Oct 30 18:44:26.860738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.860707    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 18:44:26.982291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982237    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982401    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982434    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s85\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982476    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982514    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982558    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982639    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982672    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:26.982823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:26.982729    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.083896 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.083869    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.083906    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.083925    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.083975    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084005    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084053    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084135    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084562 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084217    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084605    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s85\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084448    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084395    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084638    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.084976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084773    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.085234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.084996    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.087480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.087420    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.087909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.087887    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.098068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.098048    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.103924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.103897    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s85\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.162426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.162399    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:27.478450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.478422    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 18:44:27.479149 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:27.479123    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd75150_bfd0_40c6_ab71_9cb0e107cee8.slice/crio-da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443 WatchSource:0}: Error finding container da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443: Status 404 returned error can't find the container with id da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443
Oct 30 18:44:27.482118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.482067    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:27.482203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.482162    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:27.482243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.482199    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:27.571069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.571039    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" event={"ID":"ebd75150-bfd0-40c6-ab71-9cb0e107cee8","Type":"ContainerStarted","Data":"0bf23cc51402ccf8582c06caec8b4414273b51d859f9ed3f1b25753d99d85521"}
Oct 30 18:44:27.571181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.571116    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" event={"ID":"ebd75150-bfd0-40c6-ab71-9cb0e107cee8","Type":"ContainerStarted","Data":"da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443"}
Oct 30 18:44:27.602242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:27.602183    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podStartSLOduration=1.602161167 podStartE2EDuration="1.602161167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:27.600966337 +0000 UTC m=+687.864431305" watchObservedRunningTime="2026-10-30 18:44:27.602161167 +0000 UTC m=+687.865626115"
Oct 30 18:44:28.163109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:28.163054    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:28.164826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:28.164802    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:28.164949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:28.164849    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:29.163129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.163065    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:29.163518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.163153    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:29.463755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.463679    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 18:44:29.467685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.467668    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.477952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.477931    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-da70a5-openshift-default-dockercfg-vx9wq\""
Oct 30 18:44:29.505934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.505906    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 18:44:29.617932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.617900    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.617932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.617938    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.617977    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618139    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618210    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618288    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618308    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqm96\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618347    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.618413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.618402    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.718986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.718912    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.718986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.718954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.718986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.718977    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719024    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719219    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719359    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719389    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719447    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719475    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqm96\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719518    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719654    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719686    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.719866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.719842    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.720207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.720143    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.721706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.721687    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.721860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.721839    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.768719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.768698    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.769001 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.768976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqm96\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:29.798476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:29.798252    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:30.113631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.113598    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 18:44:30.114565 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:30.114537    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbcd5f2_48e7_4a14_a2b9_56a3e847af34.slice/crio-45e5ccd776508b9c9fac2db499e6d2e49fb23aa63bd5b8f850516f63b4e47c24 WatchSource:0}: Error finding container 45e5ccd776508b9c9fac2db499e6d2e49fb23aa63bd5b8f850516f63b4e47c24: Status 404 returned error can't find the container with id 45e5ccd776508b9c9fac2db499e6d2e49fb23aa63bd5b8f850516f63b4e47c24
Oct 30 18:44:30.117053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.117019    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:30.117179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.117133    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:30.117245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.117183    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:30.163403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.163377    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:30.163770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.163418    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:30.645201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.638197    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" event={"ID":"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34","Type":"ContainerStarted","Data":"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"}
Oct 30 18:44:30.645201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.638327    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" event={"ID":"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34","Type":"ContainerStarted","Data":"45e5ccd776508b9c9fac2db499e6d2e49fb23aa63bd5b8f850516f63b4e47c24"}
Oct 30 18:44:30.671247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.671197    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podStartSLOduration=1.671181286 podStartE2EDuration="1.671181286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:30.668795415 +0000 UTC m=+690.932260365" watchObservedRunningTime="2026-10-30 18:44:30.671181286 +0000 UTC m=+690.934646225"
Oct 30 18:44:30.804110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:30.802203    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:31.163381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:31.163349    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:31.163565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:31.163410    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:31.815238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:31.815202    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:31.815565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:31.815541    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:32.162833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.162743    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:32.162833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.162812    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:32.805112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.800454    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:32.805112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.800526    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:32.812223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.809402    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:32.851962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:32.851932    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:33.162968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:33.162883    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:33.162968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:33.162935    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:33.673822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:33.673798    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:44:33.800493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:33.800458    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:33.800856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:33.800818    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:34.163653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:34.163612    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:44:34.164027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:34.163675    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:44:34.803112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:34.802185    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:34.803112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:34.802278    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:35.168150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:35.167795    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:35.168521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:35.168367    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:35.170702 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:35.169277    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 18:44:35.800529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:35.800492    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:35.800828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:35.800797    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:36.809479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:36.809444    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 18:44:36.810056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:36.810023    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 18:44:37.923593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:37.923559    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:37.924870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:37.924831    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:37.926030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:37.926008    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 18:44:46.901710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.901677    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:44:46.905507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.905483    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.921372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.921347    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8a92cc-openshift-default-dockercfg-vj746\""
Oct 30 18:44:46.934669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.934532    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:44:46.982757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.982721    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.982950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.982778    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.982950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.982899    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983082 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983043    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983082 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983108    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983137    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983320    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983360    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:46.983457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:46.983442    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9j84\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084580    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084628    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9j84\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084760    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084790    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.084883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084833    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084887    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084922    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.084950    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.085025    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.085558    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.085594    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.085695    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.085923 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.085863    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.087900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.087875    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.087985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.087926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.117285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.117259    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.122130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.122100    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9j84\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.221051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.220966    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:47.538975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.538947    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:44:47.538975 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:47.538965    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09939065_c860_48e5_9c62_6b646046638c.slice/crio-50b102a8db703bedaff4d4137a8620225fd73767aca4673cb18ac25a9a180883 WatchSource:0}: Error finding container 50b102a8db703bedaff4d4137a8620225fd73767aca4673cb18ac25a9a180883: Status 404 returned error can't find the container with id 50b102a8db703bedaff4d4137a8620225fd73767aca4673cb18ac25a9a180883
Oct 30 18:44:47.542667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.542201    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:47.542667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.542355    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:47.542667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.542442    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:44:47.819650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.819536    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" event={"ID":"09939065-c860-48e5-9c62-6b646046638c","Type":"ContainerStarted","Data":"42f3dd9a042ce3436247644a40984886cd7aa239201e6e2d0acc6240be55eb55"}
Oct 30 18:44:47.819816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:47.819652    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" event={"ID":"09939065-c860-48e5-9c62-6b646046638c","Type":"ContainerStarted","Data":"50b102a8db703bedaff4d4137a8620225fd73767aca4673cb18ac25a9a180883"}
Oct 30 18:44:48.222043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:48.221950    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:48.223756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:48.223725    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:48.223890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:48.223778    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:49.221465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:49.221432    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:49.221676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:49.221484    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:50.221332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:50.221297    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:50.221812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:50.221354    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:51.221429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:51.221395    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:51.221918 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:51.221456    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:52.221530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:52.221500    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:52.221926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:52.221547    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:53.057717 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:44:53.057681    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:44:53.221597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:53.221564    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:44:53.222143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:53.221619    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 18:44:54.372787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:54.372741    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:54.373310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:54.373209    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:54.375078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:54.375052    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:44:54.420108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:54.420037    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podStartSLOduration=8.420018192 podStartE2EDuration="8.420018192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:47.859390135 +0000 UTC m=+708.122855094" watchObservedRunningTime="2026-10-30 18:44:54.420018192 +0000 UTC m=+714.683483140"
Oct 30 18:44:55.312427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.312387    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"]
Oct 30 18:44:55.318688 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.318663    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.324209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.324185    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:44:55.324900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.324876    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"]
Oct 30 18:44:55.325637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.325620    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:44:55.409527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.409496    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.410114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.410069    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8nc\" (UniqueName: \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.410271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.410256    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.512149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.512075    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.512546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.512519    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8nc\" (UniqueName: \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.514873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.514848    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.515273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.515245    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.515770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.515746    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.526742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.526711    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8nc\" (UniqueName: \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\") pod \"maas-api-revoke-sub-800848d5-phxnq\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.637411 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.637346    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:55.932360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:55.932326    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"]
Oct 30 18:44:55.933638 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:55.933611    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11e4613_303c_4f06_9800_f6c86e7b61fd.slice/crio-7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910 WatchSource:0}: Error finding container 7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910: Status 404 returned error can't find the container with id 7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910
Oct 30 18:44:56.205975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.205941    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"]
Oct 30 18:44:56.210942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.210917    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.224565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.224534    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"]
Oct 30 18:44:56.331340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.331308    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.331496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.331370    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qv8\" (UniqueName: \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.331496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.331484    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.432711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.432675    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.433190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.432729    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qv8\" (UniqueName: \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.433190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.432781    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.433449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.433429    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.435678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.435653    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.447080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.447033    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qv8\" (UniqueName: \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\") pod \"maas-api-revoke-sub-47df19d8-kn86m\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.529080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.529054    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:56.815517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.815488    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"]
Oct 30 18:44:56.816956 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:44:56.816932    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3759e6_80d4_4d96_bd03_1b7079099af3.slice/crio-7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff WatchSource:0}: Error finding container 7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff: Status 404 returned error can't find the container with id 7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff
Oct 30 18:44:56.879999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.879973    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m" event={"ID":"3c3759e6-80d4-4d96-bd03-1b7079099af3","Type":"ContainerStarted","Data":"7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff"}
Oct 30 18:44:56.882143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.882076    5065 generic.go:356] "Generic (PLEG): container finished" podID="b11e4613-303c-4f06-9800-f6c86e7b61fd" containerID="786f361b082082eec20679b101290fea6b791d121a70427f51e9c4bbde692964" exitCode=0
Oct 30 18:44:56.882261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.882209    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq" event={"ID":"b11e4613-303c-4f06-9800-f6c86e7b61fd","Type":"ContainerDied","Data":"786f361b082082eec20679b101290fea6b791d121a70427f51e9c4bbde692964"}
Oct 30 18:44:56.882261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:56.882243    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq" event={"ID":"b11e4613-303c-4f06-9800-f6c86e7b61fd","Type":"ContainerStarted","Data":"7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910"}
Oct 30 18:44:57.892255 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:57.892209    5065 generic.go:356] "Generic (PLEG): container finished" podID="3c3759e6-80d4-4d96-bd03-1b7079099af3" containerID="4220d6c836244db887c7a7944686c811b8d01303637054720161f0634e8ef8ba" exitCode=0
Oct 30 18:44:57.892753 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:57.892307    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m" event={"ID":"3c3759e6-80d4-4d96-bd03-1b7079099af3","Type":"ContainerDied","Data":"4220d6c836244db887c7a7944686c811b8d01303637054720161f0634e8ef8ba"}
Oct 30 18:44:58.065149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.065009    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:58.250883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.250845    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\") pod \"b11e4613-303c-4f06-9800-f6c86e7b61fd\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") "
Oct 30 18:44:58.251071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.250976    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\") pod \"b11e4613-303c-4f06-9800-f6c86e7b61fd\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") "
Oct 30 18:44:58.251138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.251066    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\" (UniqueName: \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\") pod \"b11e4613-303c-4f06-9800-f6c86e7b61fd\" (UID: \"b11e4613-303c-4f06-9800-f6c86e7b61fd\") "
Oct 30 18:44:58.251463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.251428    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca" pod "b11e4613-303c-4f06-9800-f6c86e7b61fd" (UID: "b11e4613-303c-4f06-9800-f6c86e7b61fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:44:58.251639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.251620    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:58.253364 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.253339    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca" pod "b11e4613-303c-4f06-9800-f6c86e7b61fd" (UID: "b11e4613-303c-4f06-9800-f6c86e7b61fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:44:58.253684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.253669    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc" pod "b11e4613-303c-4f06-9800-f6c86e7b61fd" (UID: "b11e4613-303c-4f06-9800-f6c86e7b61fd"). InnerVolumeSpecName "kube-api-access-ld8nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:58.352646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.352545    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8nc\" (UniqueName: \"kubernetes.io/projected/b11e4613-303c-4f06-9800-f6c86e7b61fd-kube-api-access-ld8nc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:58.352646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.352579    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e4613-303c-4f06-9800-f6c86e7b61fd-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:58.904955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.904924    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"
Oct 30 18:44:58.905441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.904927    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq" event={"ID":"b11e4613-303c-4f06-9800-f6c86e7b61fd","Type":"ContainerDied","Data":"7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910"}
Oct 30 18:44:58.905441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:58.905074    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7396491cb0db7119ae2614fdc9432d85beb8b03a5ba85261175611638aa8a910"
Oct 30 18:44:59.072324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.072290    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:59.161456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.161363    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\") pod \"3c3759e6-80d4-4d96-bd03-1b7079099af3\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") "
Oct 30 18:44:59.161456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.161414    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\") pod \"3c3759e6-80d4-4d96-bd03-1b7079099af3\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") "
Oct 30 18:44:59.161697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.161519    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\" (UniqueName: \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\") pod \"3c3759e6-80d4-4d96-bd03-1b7079099af3\" (UID: \"3c3759e6-80d4-4d96-bd03-1b7079099af3\") "
Oct 30 18:44:59.161799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.161767    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca" pod "3c3759e6-80d4-4d96-bd03-1b7079099af3" (UID: "3c3759e6-80d4-4d96-bd03-1b7079099af3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:44:59.162069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.162039    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:59.163742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.163719    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8" pod "3c3759e6-80d4-4d96-bd03-1b7079099af3" (UID: "3c3759e6-80d4-4d96-bd03-1b7079099af3"). InnerVolumeSpecName "kube-api-access-k9qv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:59.163907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.163849    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca" pod "3c3759e6-80d4-4d96-bd03-1b7079099af3" (UID: "3c3759e6-80d4-4d96-bd03-1b7079099af3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:44:59.262768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.262721    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3759e6-80d4-4d96-bd03-1b7079099af3-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:59.262768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.262760    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qv8\" (UniqueName: \"kubernetes.io/projected/3c3759e6-80d4-4d96-bd03-1b7079099af3-kube-api-access-k9qv8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:59.916151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.916119    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"
Oct 30 18:44:59.916151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.916128    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m" event={"ID":"3c3759e6-80d4-4d96-bd03-1b7079099af3","Type":"ContainerDied","Data":"7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff"}
Oct 30 18:44:59.916705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:44:59.916178    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7b77a09f6785671cbcf8ae339b3508a405f8b9be4573292cd92446309ee6ff"
Oct 30 18:45:00.169610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.169525    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:00.170196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170169    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3c3759e6-80d4-4d96-bd03-1b7079099af3" containerName="revoke-keys"
Oct 30 18:45:00.170196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170197    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3759e6-80d4-4d96-bd03-1b7079099af3" containerName="revoke-keys"
Oct 30 18:45:00.170318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170261    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b11e4613-303c-4f06-9800-f6c86e7b61fd" containerName="revoke-keys"
Oct 30 18:45:00.170318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170267    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11e4613-303c-4f06-9800-f6c86e7b61fd" containerName="revoke-keys"
Oct 30 18:45:00.170379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170369    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3759e6-80d4-4d96-bd03-1b7079099af3" containerName="revoke-keys"
Oct 30 18:45:00.170379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.170386    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11e4613-303c-4f06-9800-f6c86e7b61fd" containerName="revoke-keys"
Oct 30 18:45:00.174069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.174046    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:00.174246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.174211    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.177942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.177916    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:00.178068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.177998    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.181858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.181832    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.193838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.193815    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:45:00.207112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.205688    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:00.212670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.212642    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:00.215099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.215046    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:00.232826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.232786    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:00.238308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.238152    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.252895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.252362    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:00.271851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.271810    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.272180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.271857    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8cv\" (UniqueName: \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.272180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.271894    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.272180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.272050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k487c\" (UniqueName: \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.272180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.272134    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.272357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.272214    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl77g\" (UniqueName: \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.272357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.272286    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42r8\" (UniqueName: \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.272449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.272355    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.373865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.373826    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.374055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.373876    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8cv\" (UniqueName: \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.374055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.373919    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.374055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.373952    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k487c\" (UniqueName: \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.374055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.373996    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.374282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374069    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl77g\" (UniqueName: \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.374282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374192    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42r8\" (UniqueName: \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.374282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374262    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.374742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374713    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.374785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374761    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.374839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374823    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.374906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.374886    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.398306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.398272    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42r8\" (UniqueName: \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.398963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.398863    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k487c\" (UniqueName: \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.400490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.400463    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl77g\" (UniqueName: \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.400665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.400646    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8cv\" (UniqueName: \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.490689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.490574    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:00.502268 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.502234    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:00.516184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.516155    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:00.561851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.561811    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:00.812828 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:00.812759    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534a2020_6546_4861_9a16_da893819321e.slice/crio-8bd008740a5d40a8c3b82659ba07d0f00ca29571b4c19e60c1e788f1b8ceb3ff WatchSource:0}: Error finding container 8bd008740a5d40a8c3b82659ba07d0f00ca29571b4c19e60c1e788f1b8ceb3ff: Status 404 returned error can't find the container with id 8bd008740a5d40a8c3b82659ba07d0f00ca29571b4c19e60c1e788f1b8ceb3ff
Oct 30 18:45:00.814335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.814241    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:00.830460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.828824    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:00.877587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.877562    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:00.880251 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:00.880214    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4ab1a1_046c_4034_8e9e_7e4be4803a6d.slice/crio-f138bf1448a4f40e5ecfe1f41094361af99eefc607769d1aee13f9561e2c3fdb WatchSource:0}: Error finding container f138bf1448a4f40e5ecfe1f41094361af99eefc607769d1aee13f9561e2c3fdb: Status 404 returned error can't find the container with id f138bf1448a4f40e5ecfe1f41094361af99eefc607769d1aee13f9561e2c3fdb
Oct 30 18:45:00.937498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.937435    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerStarted","Data":"8bd008740a5d40a8c3b82659ba07d0f00ca29571b4c19e60c1e788f1b8ceb3ff"}
Oct 30 18:45:00.941026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.940650    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerStarted","Data":"484e8f80329ab115026384a2e35b8aef02f679773143926460c14c8b2a5fbfbc"}
Oct 30 18:45:00.942533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.942502    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerStarted","Data":"f138bf1448a4f40e5ecfe1f41094361af99eefc607769d1aee13f9561e2c3fdb"}
Oct 30 18:45:00.944550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:00.944527    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:01.951685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.951644    5065 generic.go:356] "Generic (PLEG): container finished" podID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerID="6963045bb2e6bf65de888ce3685807cde95a6746ac673736a04a020e7eb503a8" exitCode=60
Oct 30 18:45:01.952237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.951720    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerDied","Data":"6963045bb2e6bf65de888ce3685807cde95a6746ac673736a04a020e7eb503a8"}
Oct 30 18:45:01.952237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.952009    5065 scope.go:122] "RemoveContainer" containerID="6963045bb2e6bf65de888ce3685807cde95a6746ac673736a04a020e7eb503a8"
Oct 30 18:45:01.953944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.953918    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerStarted","Data":"c1c6d7814f409bfb74eedde44b232320a871c4a3e94248acde9ea43cd2da0f48"}
Oct 30 18:45:01.957010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.956548    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerStarted","Data":"edf7918df3efd9ac6ee6004e9f8d80b328676e9a4e85e391f4e762b631dc5b3c"}
Oct 30 18:45:01.957176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.957034    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerStarted","Data":"67a64cb718b85e413a305a3ee697d9866c1fd50f16eb99e3a86857294b1562bf"}
Oct 30 18:45:01.959180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:01.959145    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerStarted","Data":"bb5c57e4855aed68683b39dbbf3c4b89d29a3e3de650804671b2d0bd1f9367cb"}
Oct 30 18:45:02.054556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.053003    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" podStartSLOduration=2.052980474 podStartE2EDuration="2.052980474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.047998417 +0000 UTC m=+722.311463382" watchObservedRunningTime="2026-10-30 18:45:02.052980474 +0000 UTC m=+722.316445424"
Oct 30 18:45:02.070762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.070696    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" podStartSLOduration=2.070678803 podStartE2EDuration="2.070678803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.023013876 +0000 UTC m=+722.286478837" watchObservedRunningTime="2026-10-30 18:45:02.070678803 +0000 UTC m=+722.334143752"
Oct 30 18:45:02.077861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.077789    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" podStartSLOduration=2.07776744 podStartE2EDuration="2.07776744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.069899127 +0000 UTC m=+722.333364076" watchObservedRunningTime="2026-10-30 18:45:02.07776744 +0000 UTC m=+722.341232390"
Oct 30 18:45:02.977621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.977528    5065 generic.go:356] "Generic (PLEG): container finished" podID="534a2020-6546-4861-9a16-da893819321e" containerID="bb5c57e4855aed68683b39dbbf3c4b89d29a3e3de650804671b2d0bd1f9367cb" exitCode=60
Oct 30 18:45:02.978112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.977638    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerDied","Data":"bb5c57e4855aed68683b39dbbf3c4b89d29a3e3de650804671b2d0bd1f9367cb"}
Oct 30 18:45:02.978112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.977900    5065 scope.go:122] "RemoveContainer" containerID="bb5c57e4855aed68683b39dbbf3c4b89d29a3e3de650804671b2d0bd1f9367cb"
Oct 30 18:45:02.986542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.986114    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerStarted","Data":"ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"}
Oct 30 18:45:02.990223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.990197    5065 generic.go:356] "Generic (PLEG): container finished" podID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerID="c1c6d7814f409bfb74eedde44b232320a871c4a3e94248acde9ea43cd2da0f48" exitCode=60
Oct 30 18:45:02.990357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.990337    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerDied","Data":"c1c6d7814f409bfb74eedde44b232320a871c4a3e94248acde9ea43cd2da0f48"}
Oct 30 18:45:02.990542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.990518    5065 scope.go:122] "RemoveContainer" containerID="c1c6d7814f409bfb74eedde44b232320a871c4a3e94248acde9ea43cd2da0f48"
Oct 30 18:45:02.992794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.992763    5065 generic.go:356] "Generic (PLEG): container finished" podID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerID="edf7918df3efd9ac6ee6004e9f8d80b328676e9a4e85e391f4e762b631dc5b3c" exitCode=60
Oct 30 18:45:02.992900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.992835    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerDied","Data":"edf7918df3efd9ac6ee6004e9f8d80b328676e9a4e85e391f4e762b631dc5b3c"}
Oct 30 18:45:02.993186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:02.993164    5065 scope.go:122] "RemoveContainer" containerID="edf7918df3efd9ac6ee6004e9f8d80b328676e9a4e85e391f4e762b631dc5b3c"
Oct 30 18:45:03.217711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:03.217661    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" podStartSLOduration=3.217645044 podStartE2EDuration="3.217645044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.153421268 +0000 UTC m=+723.416886218" watchObservedRunningTime="2026-10-30 18:45:03.217645044 +0000 UTC m=+723.481109991"
Oct 30 18:45:04.004064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.004027    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerStarted","Data":"2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"}
Oct 30 18:45:04.006339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.006310    5065 generic.go:356] "Generic (PLEG): container finished" podID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerID="ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c" exitCode=60
Oct 30 18:45:04.006460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.006398    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerDied","Data":"ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"}
Oct 30 18:45:04.006460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.006455    5065 scope.go:122] "RemoveContainer" containerID="6963045bb2e6bf65de888ce3685807cde95a6746ac673736a04a020e7eb503a8"
Oct 30 18:45:04.006618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.006594    5065 scope.go:122] "RemoveContainer" containerID="ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"
Oct 30 18:45:04.006890 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:04.006871    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm_odh-ai-gateway-infra(3bbeaae0-c333-4eb1-8721-2a3c3cd02db2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2"
Oct 30 18:45:04.009845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.009818    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerStarted","Data":"dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"}
Oct 30 18:45:04.012173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.012149    5065 generic.go:356] "Generic (PLEG): container finished" podID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerID="764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566" exitCode=60
Oct 30 18:45:04.012289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.012202    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerDied","Data":"764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566"}
Oct 30 18:45:04.012446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.012430    5065 scope.go:122] "RemoveContainer" containerID="764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566"
Oct 30 18:45:04.012668 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:04.012648    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8_odh-ai-gateway-infra(5485716f-aaf9-409b-ac95-e6a9959acee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0"
Oct 30 18:45:04.054391 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:04.054364    5065 scope.go:122] "RemoveContainer" containerID="edf7918df3efd9ac6ee6004e9f8d80b328676e9a4e85e391f4e762b631dc5b3c"
Oct 30 18:45:05.021474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.021438    5065 generic.go:356] "Generic (PLEG): container finished" podID="534a2020-6546-4861-9a16-da893819321e" containerID="2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819" exitCode=60
Oct 30 18:45:05.021995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.021523    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerDied","Data":"2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"}
Oct 30 18:45:05.021995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.021592    5065 scope.go:122] "RemoveContainer" containerID="bb5c57e4855aed68683b39dbbf3c4b89d29a3e3de650804671b2d0bd1f9367cb"
Oct 30 18:45:05.021995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.021799    5065 scope.go:122] "RemoveContainer" containerID="2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"
Oct 30 18:45:05.022176 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:05.022129    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977_odh-ai-gateway-infra(534a2020-6546-4861-9a16-da893819321e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" podUID="534a2020-6546-4861-9a16-da893819321e"
Oct 30 18:45:05.024039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.024022    5065 scope.go:122] "RemoveContainer" containerID="ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"
Oct 30 18:45:05.024353 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:05.024333    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm_odh-ai-gateway-infra(3bbeaae0-c333-4eb1-8721-2a3c3cd02db2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2"
Oct 30 18:45:05.025917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.025895    5065 generic.go:356] "Generic (PLEG): container finished" podID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerID="dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d" exitCode=60
Oct 30 18:45:05.026048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.025947    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerDied","Data":"dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"}
Oct 30 18:45:05.026286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.026264    5065 scope.go:122] "RemoveContainer" containerID="dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"
Oct 30 18:45:05.026777 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:05.026748    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl_odh-ai-gateway-infra(9e4ab1a1-046c-4034-8e9e-7e4be4803a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d"
Oct 30 18:45:05.028214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.028184    5065 scope.go:122] "RemoveContainer" containerID="764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566"
Oct 30 18:45:05.028399 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:05.028380    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8_odh-ai-gateway-infra(5485716f-aaf9-409b-ac95-e6a9959acee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0"
Oct 30 18:45:05.035481 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:05.035464    5065 scope.go:122] "RemoveContainer" containerID="c1c6d7814f409bfb74eedde44b232320a871c4a3e94248acde9ea43cd2da0f48"
Oct 30 18:45:06.036474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:06.036448    5065 scope.go:122] "RemoveContainer" containerID="2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"
Oct 30 18:45:06.036937 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:06.036799    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977_odh-ai-gateway-infra(534a2020-6546-4861-9a16-da893819321e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" podUID="534a2020-6546-4861-9a16-da893819321e"
Oct 30 18:45:06.038350 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:06.038328    5065 scope.go:122] "RemoveContainer" containerID="dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"
Oct 30 18:45:06.038575 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:06.038550    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl_odh-ai-gateway-infra(9e4ab1a1-046c-4034-8e9e-7e4be4803a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d"
Oct 30 18:45:15.891052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:15.891013    5065 scope.go:122] "RemoveContainer" containerID="dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"
Oct 30 18:45:16.882543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:16.882516    5065 scope.go:122] "RemoveContainer" containerID="ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"
Oct 30 18:45:17.164368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:17.164335    5065 generic.go:356] "Generic (PLEG): container finished" podID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerID="3dcd5584a6bbd6bfa8eec7bb8d7f95d352c87ad801bde6b03bbc9dff90cbca7c" exitCode=60
Oct 30 18:45:17.164845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:17.164402    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerDied","Data":"3dcd5584a6bbd6bfa8eec7bb8d7f95d352c87ad801bde6b03bbc9dff90cbca7c"}
Oct 30 18:45:17.164845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:17.164460    5065 scope.go:122] "RemoveContainer" containerID="dd42c48753c0edd382352d5a7c93cce13fc691606c5088e90fe717f7eb6a213d"
Oct 30 18:45:17.164952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:17.164852    5065 scope.go:122] "RemoveContainer" containerID="3dcd5584a6bbd6bfa8eec7bb8d7f95d352c87ad801bde6b03bbc9dff90cbca7c"
Oct 30 18:45:17.165209 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:17.165183    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl_odh-ai-gateway-infra(9e4ab1a1-046c-4034-8e9e-7e4be4803a6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d"
Oct 30 18:45:17.893445 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:17.893415    5065 scope.go:122] "RemoveContainer" containerID="2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"
Oct 30 18:45:18.173270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.173236    5065 generic.go:356] "Generic (PLEG): container finished" podID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerID="20c9417917189554edcf80a1ae7cbb07d2f56bb82af592ca30f4dd078f2698d2" exitCode=60
Oct 30 18:45:18.173741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.173311    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerDied","Data":"20c9417917189554edcf80a1ae7cbb07d2f56bb82af592ca30f4dd078f2698d2"}
Oct 30 18:45:18.173741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.173379    5065 scope.go:122] "RemoveContainer" containerID="ef572ed6fe78d27463c875ad0fb8e79ca7f982a08641eed863a337a980a9d62c"
Oct 30 18:45:18.173854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.173748    5065 scope.go:122] "RemoveContainer" containerID="20c9417917189554edcf80a1ae7cbb07d2f56bb82af592ca30f4dd078f2698d2"
Oct 30 18:45:18.174045 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:18.174016    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm_odh-ai-gateway-infra(3bbeaae0-c333-4eb1-8721-2a3c3cd02db2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2"
Oct 30 18:45:18.205871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.205852    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:18.372597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.372570    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:18.438062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.438032    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\" (UniqueName: \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\") pod \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") "
Oct 30 18:45:18.438229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.438107    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\") pod \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\" (UID: \"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d\") "
Oct 30 18:45:18.438682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.438651    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca" pod "9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" (UID: "9e4ab1a1-046c-4034-8e9e-7e4be4803a6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:18.440854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.440825    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g" pod "9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" (UID: "9e4ab1a1-046c-4034-8e9e-7e4be4803a6d"). InnerVolumeSpecName "kube-api-access-cl77g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:18.539511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.539483    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:18.539617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.539514    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl77g\" (UniqueName: \"kubernetes.io/projected/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d-kube-api-access-cl77g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:18.882378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:18.882343    5065 scope.go:122] "RemoveContainer" containerID="764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566"
Oct 30 18:45:19.203134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.203070    5065 generic.go:356] "Generic (PLEG): container finished" podID="534a2020-6546-4861-9a16-da893819321e" containerID="aa098bd226abbb8efa7fa165861762baeff064b7f1f42d718a12e3d5a08d941b" exitCode=60
Oct 30 18:45:19.203480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.203241    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerDied","Data":"aa098bd226abbb8efa7fa165861762baeff064b7f1f42d718a12e3d5a08d941b"}
Oct 30 18:45:19.203480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.203346    5065 scope.go:122] "RemoveContainer" containerID="2d8856aff6aab2ec2affffb419e73166b3b1d07baed035dd80db4293596ce819"
Oct 30 18:45:19.203919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.203887    5065 scope.go:122] "RemoveContainer" containerID="aa098bd226abbb8efa7fa165861762baeff064b7f1f42d718a12e3d5a08d941b"
Oct 30 18:45:19.206079 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:19.206046    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977_odh-ai-gateway-infra(534a2020-6546-4861-9a16-da893819321e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" podUID="534a2020-6546-4861-9a16-da893819321e"
Oct 30 18:45:19.211239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.211215    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl" event={"ID":"9e4ab1a1-046c-4034-8e9e-7e4be4803a6d","Type":"ContainerDied","Data":"f138bf1448a4f40e5ecfe1f41094361af99eefc607769d1aee13f9561e2c3fdb"}
Oct 30 18:45:19.211401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.211328    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"
Oct 30 18:45:19.213031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.213005    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:19.229940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.229844    5065 scope.go:122] "RemoveContainer" containerID="3dcd5584a6bbd6bfa8eec7bb8d7f95d352c87ad801bde6b03bbc9dff90cbca7c"
Oct 30 18:45:19.250009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.249941    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:19.254162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.254141    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889765-srwvl"]
Oct 30 18:45:19.429557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.429529    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:19.451819 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.451793    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\" (UniqueName: \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\") pod \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") "
Oct 30 18:45:19.451943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.451891    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\") pod \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\" (UID: \"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2\") "
Oct 30 18:45:19.452384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.452357    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca" pod "3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" (UID: "3bbeaae0-c333-4eb1-8721-2a3c3cd02db2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:19.454781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.454752    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c" pod "3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" (UID: "3bbeaae0-c333-4eb1-8721-2a3c3cd02db2"). InnerVolumeSpecName "kube-api-access-k487c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:19.553761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.553732    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k487c\" (UniqueName: \"kubernetes.io/projected/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-kube-api-access-k487c\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:19.553761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.553761    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:19.893569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:19.893538    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" path="/var/lib/kubelet/pods/9e4ab1a1-046c-4034-8e9e-7e4be4803a6d/volumes"
Oct 30 18:45:20.221525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.221441    5065 generic.go:356] "Generic (PLEG): container finished" podID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerID="be0893cb84534449b87ad11d56732bd5c3b76242bc545ef90b94c7325ab6a551" exitCode=60
Oct 30 18:45:20.221525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.221509    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerDied","Data":"be0893cb84534449b87ad11d56732bd5c3b76242bc545ef90b94c7325ab6a551"}
Oct 30 18:45:20.222029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.221567    5065 scope.go:122] "RemoveContainer" containerID="764bfb6dcf2c5c6fac385a8f3f1894d4b9feae419495879557c0948f062c3566"
Oct 30 18:45:20.222029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.221929    5065 scope.go:122] "RemoveContainer" containerID="be0893cb84534449b87ad11d56732bd5c3b76242bc545ef90b94c7325ab6a551"
Oct 30 18:45:20.222254 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:20.222230    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8_odh-ai-gateway-infra(5485716f-aaf9-409b-ac95-e6a9959acee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0"
Oct 30 18:45:20.225120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.225032    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"
Oct 30 18:45:20.225120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.225061    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm" event={"ID":"3bbeaae0-c333-4eb1-8721-2a3c3cd02db2","Type":"ContainerDied","Data":"484e8f80329ab115026384a2e35b8aef02f679773143926460c14c8b2a5fbfbc"}
Oct 30 18:45:20.235762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.235744    5065 scope.go:122] "RemoveContainer" containerID="20c9417917189554edcf80a1ae7cbb07d2f56bb82af592ca30f4dd078f2698d2"
Oct 30 18:45:20.263734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.263710    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:20.345079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.345018    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:20.348061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.348035    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889765-rh6wm"]
Oct 30 18:45:20.416931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.416907    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:20.465158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.465132    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\") pod \"534a2020-6546-4861-9a16-da893819321e\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") "
Oct 30 18:45:20.465339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.465257    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\" (UniqueName: \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\") pod \"534a2020-6546-4861-9a16-da893819321e\" (UID: \"534a2020-6546-4861-9a16-da893819321e\") "
Oct 30 18:45:20.465444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.465425    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca" pod "534a2020-6546-4861-9a16-da893819321e" (UID: "534a2020-6546-4861-9a16-da893819321e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:20.465709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.465690    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a2020-6546-4861-9a16-da893819321e-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:20.467579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.467558    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8" pod "534a2020-6546-4861-9a16-da893819321e" (UID: "534a2020-6546-4861-9a16-da893819321e"). InnerVolumeSpecName "kube-api-access-f42r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:20.567042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:20.567003    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f42r8\" (UniqueName: \"kubernetes.io/projected/534a2020-6546-4861-9a16-da893819321e-kube-api-access-f42r8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:21.092265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092232    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"]
Oct 30 18:45:21.092962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092933    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.092962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092959    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092972    5065 cpu_manager.go:414] "containerMap: removing container" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092977    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.092999    5065 cpu_manager.go:414] "containerMap: removing container" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093009    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093029    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093036    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093045    5065 cpu_manager.go:414] "containerMap: removing container" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093052    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093076    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093098    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093136    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093143    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093261    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093273    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093290    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093303    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093311    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a2020-6546-4861-9a16-da893819321e" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093319    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:21.093814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.093326    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:21.097131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.097112    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.113173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.113150    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"]
Oct 30 18:45:21.122830 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.122813    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:45:21.173950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.173923    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.174114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.173961    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.174114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.174078    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6s42\" (UniqueName: \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.233608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.233587    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"
Oct 30 18:45:21.233978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.233603    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977" event={"ID":"534a2020-6546-4861-9a16-da893819321e","Type":"ContainerDied","Data":"8bd008740a5d40a8c3b82659ba07d0f00ca29571b4c19e60c1e788f1b8ceb3ff"}
Oct 30 18:45:21.233978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.233667    5065 scope.go:122] "RemoveContainer" containerID="aa098bd226abbb8efa7fa165861762baeff064b7f1f42d718a12e3d5a08d941b"
Oct 30 18:45:21.260542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.260519    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:21.263318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.263301    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889765-xz977"]
Oct 30 18:45:21.275745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.275720    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6s42\" (UniqueName: \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.275866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.275848    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.275915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.275902    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.276525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.276506    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.278243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.278223    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.308793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.308769    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6s42\" (UniqueName: \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\") pod \"maas-api-revoke-sub-4255e13f-p87tk\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.319873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.319854    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:21.410936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.410911    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:21.473839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.473549    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:21.581611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.581582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\") pod \"5485716f-aaf9-409b-ac95-e6a9959acee0\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") "
Oct 30 18:45:21.581764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.581644    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\" (UniqueName: \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\") pod \"5485716f-aaf9-409b-ac95-e6a9959acee0\" (UID: \"5485716f-aaf9-409b-ac95-e6a9959acee0\") "
Oct 30 18:45:21.581922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.581899    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca" pod "5485716f-aaf9-409b-ac95-e6a9959acee0" (UID: "5485716f-aaf9-409b-ac95-e6a9959acee0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:21.582161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.582144    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5485716f-aaf9-409b-ac95-e6a9959acee0-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:21.583810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.583789    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv" pod "5485716f-aaf9-409b-ac95-e6a9959acee0" (UID: "5485716f-aaf9-409b-ac95-e6a9959acee0"). InnerVolumeSpecName "kube-api-access-sd8cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:21.683739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.683677    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8cv\" (UniqueName: \"kubernetes.io/projected/5485716f-aaf9-409b-ac95-e6a9959acee0-kube-api-access-sd8cv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:21.706124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.706098    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"]
Oct 30 18:45:21.707030 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:21.707006    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64184c22_bd6b_41c4_bba6_38c89c3ba6fd.slice/crio-e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11 WatchSource:0}: Error finding container e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11: Status 404 returned error can't find the container with id e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11
Oct 30 18:45:21.905468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.905440    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" path="/var/lib/kubelet/pods/3bbeaae0-c333-4eb1-8721-2a3c3cd02db2/volumes"
Oct 30 18:45:21.905833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:21.905820    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534a2020-6546-4861-9a16-da893819321e" path="/var/lib/kubelet/pods/534a2020-6546-4861-9a16-da893819321e/volumes"
Oct 30 18:45:22.245780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.245699    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"
Oct 30 18:45:22.245780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.245741    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8" event={"ID":"5485716f-aaf9-409b-ac95-e6a9959acee0","Type":"ContainerDied","Data":"67a64cb718b85e413a305a3ee697d9866c1fd50f16eb99e3a86857294b1562bf"}
Oct 30 18:45:22.246321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.245808    5065 scope.go:122] "RemoveContainer" containerID="be0893cb84534449b87ad11d56732bd5c3b76242bc545ef90b94c7325ab6a551"
Oct 30 18:45:22.248811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.248784    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk" event={"ID":"64184c22-bd6b-41c4-bba6-38c89c3ba6fd","Type":"ContainerStarted","Data":"fe6cdd7661b4aa889190dcd5a7cb596b4061b51e2f4be7737e3ae530c46eea3a"}
Oct 30 18:45:22.248931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.248840    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk" event={"ID":"64184c22-bd6b-41c4-bba6-38c89c3ba6fd","Type":"ContainerStarted","Data":"e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11"}
Oct 30 18:45:22.271615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.271587    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:22.276262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.276240    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889765-mx2c8"]
Oct 30 18:45:22.299177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:22.299135    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk" podStartSLOduration=1.299124329 podStartE2EDuration="1.299124329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:22.298469639 +0000 UTC m=+742.561934586" watchObservedRunningTime="2026-10-30 18:45:22.299124329 +0000 UTC m=+742.562589283"
Oct 30 18:45:23.066785 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:23.066743    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:45:23.261221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:23.261188    5065 generic.go:356] "Generic (PLEG): container finished" podID="64184c22-bd6b-41c4-bba6-38c89c3ba6fd" containerID="fe6cdd7661b4aa889190dcd5a7cb596b4061b51e2f4be7737e3ae530c46eea3a" exitCode=0
Oct 30 18:45:23.261667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:23.261270    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk" event={"ID":"64184c22-bd6b-41c4-bba6-38c89c3ba6fd","Type":"ContainerDied","Data":"fe6cdd7661b4aa889190dcd5a7cb596b4061b51e2f4be7737e3ae530c46eea3a"}
Oct 30 18:45:23.890759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:23.890720    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" path="/var/lib/kubelet/pods/5485716f-aaf9-409b-ac95-e6a9959acee0/volumes"
Oct 30 18:45:24.445929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.445909    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:24.513781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.513740    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\" (UniqueName: \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\") pod \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") "
Oct 30 18:45:24.513984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.513967    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\") pod \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") "
Oct 30 18:45:24.514057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.514026    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\") pod \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\" (UID: \"64184c22-bd6b-41c4-bba6-38c89c3ba6fd\") "
Oct 30 18:45:24.514411 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.514379    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca" pod "64184c22-bd6b-41c4-bba6-38c89c3ba6fd" (UID: "64184c22-bd6b-41c4-bba6-38c89c3ba6fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:24.516216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.516188    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42" pod "64184c22-bd6b-41c4-bba6-38c89c3ba6fd" (UID: "64184c22-bd6b-41c4-bba6-38c89c3ba6fd"). InnerVolumeSpecName "kube-api-access-g6s42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:24.516479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.516460    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca" pod "64184c22-bd6b-41c4-bba6-38c89c3ba6fd" (UID: "64184c22-bd6b-41c4-bba6-38c89c3ba6fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:24.615303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.615279    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:24.615303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.615303    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:24.615426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:24.615313    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6s42\" (UniqueName: \"kubernetes.io/projected/64184c22-bd6b-41c4-bba6-38c89c3ba6fd-kube-api-access-g6s42\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:25.290702 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:25.290653    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk" event={"ID":"64184c22-bd6b-41c4-bba6-38c89c3ba6fd","Type":"ContainerDied","Data":"e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11"}
Oct 30 18:45:25.290885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:25.290726    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60a79f6e62ca76ef86898d0ec2f47185eb0fec6b4d408cbc7407918d4219a11"
Oct 30 18:45:25.290885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:25.290802    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"
Oct 30 18:45:27.280926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.280889    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"]
Oct 30 18:45:27.281744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281724    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281745    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281770    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281778    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281801    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281808    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281821    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281827    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281837    5065 cpu_manager.go:414] "containerMap: removing container" podUID="64184c22-bd6b-41c4-bba6-38c89c3ba6fd" containerName="revoke-keys"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281845    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64184c22-bd6b-41c4-bba6-38c89c3ba6fd" containerName="revoke-keys"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281864    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:27.281911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.281870    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:27.282471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.282052    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.282471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.282067    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbeaae0-c333-4eb1-8721-2a3c3cd02db2" containerName="cleanup"
Oct 30 18:45:27.282471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.282099    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4ab1a1-046c-4034-8e9e-7e4be4803a6d" containerName="cleanup"
Oct 30 18:45:27.282471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.282113    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="64184c22-bd6b-41c4-bba6-38c89c3ba6fd" containerName="revoke-keys"
Oct 30 18:45:27.282471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.282121    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:27.289107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.286165    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.299033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.299013    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:45:27.302179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.302160    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:45:27.304075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.304053    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"]
Oct 30 18:45:27.370385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.370356    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.370516 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.370409    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.370606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.370585    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgql\" (UniqueName: \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.471424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.471396    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.471575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.471440    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.471575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.471524    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgql\" (UniqueName: \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.472006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.471975    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.474470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.474439    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.485347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.485324    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgql\" (UniqueName: \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\") pod \"maas-api-revoke-sub-f91fc570-xlrrq\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.618575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.618498    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:27.924387 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:27.924349    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f337a44_f9c4_4377_9480_910fd1999dbe.slice/crio-4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c WatchSource:0}: Error finding container 4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c: Status 404 returned error can't find the container with id 4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c
Oct 30 18:45:27.926158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:27.926132    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"]
Oct 30 18:45:28.320744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:28.320715    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq" event={"ID":"9f337a44-f9c4-4377-9480-910fd1999dbe","Type":"ContainerStarted","Data":"78ea5465bafd5a127692d317439930197dd9c6327372368f50cf966d32c68f92"}
Oct 30 18:45:28.321124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:28.320800    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq" event={"ID":"9f337a44-f9c4-4377-9480-910fd1999dbe","Type":"ContainerStarted","Data":"4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c"}
Oct 30 18:45:28.369851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:28.369783    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq" podStartSLOduration=1.369766633 podStartE2EDuration="1.369766633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:28.365357894 +0000 UTC m=+748.628822843" watchObservedRunningTime="2026-10-30 18:45:28.369766633 +0000 UTC m=+748.633231581"
Oct 30 18:45:29.341923 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:29.341887    5065 generic.go:356] "Generic (PLEG): container finished" podID="9f337a44-f9c4-4377-9480-910fd1999dbe" containerID="78ea5465bafd5a127692d317439930197dd9c6327372368f50cf966d32c68f92" exitCode=0
Oct 30 18:45:29.342631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:29.342605    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq" event={"ID":"9f337a44-f9c4-4377-9480-910fd1999dbe","Type":"ContainerDied","Data":"78ea5465bafd5a127692d317439930197dd9c6327372368f50cf966d32c68f92"}
Oct 30 18:45:30.611980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.611916    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:30.658175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.658147    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\" (UniqueName: \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\") pod \"9f337a44-f9c4-4377-9480-910fd1999dbe\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") "
Oct 30 18:45:30.658358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.658224    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\") pod \"9f337a44-f9c4-4377-9480-910fd1999dbe\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") "
Oct 30 18:45:30.658358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.658310    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\") pod \"9f337a44-f9c4-4377-9480-910fd1999dbe\" (UID: \"9f337a44-f9c4-4377-9480-910fd1999dbe\") "
Oct 30 18:45:30.658904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.658872    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca" pod "9f337a44-f9c4-4377-9480-910fd1999dbe" (UID: "9f337a44-f9c4-4377-9480-910fd1999dbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:30.660800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.660776    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca" pod "9f337a44-f9c4-4377-9480-910fd1999dbe" (UID: "9f337a44-f9c4-4377-9480-910fd1999dbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:30.661404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.661384    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql" pod "9f337a44-f9c4-4377-9480-910fd1999dbe" (UID: "9f337a44-f9c4-4377-9480-910fd1999dbe"). InnerVolumeSpecName "kube-api-access-8vgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:30.760410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.760380    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgql\" (UniqueName: \"kubernetes.io/projected/9f337a44-f9c4-4377-9480-910fd1999dbe-kube-api-access-8vgql\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:30.760410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.760409    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:30.760410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:30.760419    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f337a44-f9c4-4377-9480-910fd1999dbe-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:31.365204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:31.365172    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"
Oct 30 18:45:31.365371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:31.365224    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq" event={"ID":"9f337a44-f9c4-4377-9480-910fd1999dbe","Type":"ContainerDied","Data":"4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c"}
Oct 30 18:45:31.365371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:31.365268    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8a69ae852140b5f89998c7f3b4e6ecd0990cddaa8f5598e7cb3b5d10d50f4c"
Oct 30 18:45:37.546112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.542269    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"]
Oct 30 18:45:37.546112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.543111    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9f337a44-f9c4-4377-9480-910fd1999dbe" containerName="revoke-keys"
Oct 30 18:45:37.546112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.543129    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f337a44-f9c4-4377-9480-910fd1999dbe" containerName="revoke-keys"
Oct 30 18:45:37.546112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.543276    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f337a44-f9c4-4377-9480-910fd1999dbe" containerName="revoke-keys"
Oct 30 18:45:37.546112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.543291    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5485716f-aaf9-409b-ac95-e6a9959acee0" containerName="cleanup"
Oct 30 18:45:37.568205 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.568177    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"]
Oct 30 18:45:37.568432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.568298    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.575185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.575164    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:45:37.577640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.576552    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:45:37.677685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.677651    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.702346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.702315    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4f5\" (UniqueName: \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.702682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.702662    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.805765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.805675    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.810146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.810117    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.810409 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.810391    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4f5\" (UniqueName: \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.815837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.815815    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.859110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.858938    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4f5\" (UniqueName: \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.862492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.862470    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4dc494-jbkvn\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:37.952267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:37.952237    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:38.360224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:38.360188    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"]
Oct 30 18:45:38.360760 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:38.360736    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2760b4_cdc3_42b7_97d7_56ef8c51c337.slice/crio-3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70 WatchSource:0}: Error finding container 3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70: Status 404 returned error can't find the container with id 3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70
Oct 30 18:45:38.445476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:38.445453    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn" event={"ID":"ac2760b4-cdc3-42b7-97d7-56ef8c51c337","Type":"ContainerStarted","Data":"3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70"}
Oct 30 18:45:39.454134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:39.454074    5065 generic.go:356] "Generic (PLEG): container finished" podID="ac2760b4-cdc3-42b7-97d7-56ef8c51c337" containerID="0bdec7d8a3d708f9e7674f1de8b5461414f799448fcb72edc2d3dfa4b0bfe674" exitCode=0
Oct 30 18:45:39.454533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:39.454164    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn" event={"ID":"ac2760b4-cdc3-42b7-97d7-56ef8c51c337","Type":"ContainerDied","Data":"0bdec7d8a3d708f9e7674f1de8b5461414f799448fcb72edc2d3dfa4b0bfe674"}
Oct 30 18:45:40.628575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.628545    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:40.772200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.772169    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\") pod \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") "
Oct 30 18:45:40.772352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.772255    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\") pod \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") "
Oct 30 18:45:40.772352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.772289    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\" (UniqueName: \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\") pod \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\" (UID: \"ac2760b4-cdc3-42b7-97d7-56ef8c51c337\") "
Oct 30 18:45:40.772628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.772607    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca" pod "ac2760b4-cdc3-42b7-97d7-56ef8c51c337" (UID: "ac2760b4-cdc3-42b7-97d7-56ef8c51c337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:40.772691 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.772672    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:40.774523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.774499    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca" pod "ac2760b4-cdc3-42b7-97d7-56ef8c51c337" (UID: "ac2760b4-cdc3-42b7-97d7-56ef8c51c337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:40.774663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.774644    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5" pod "ac2760b4-cdc3-42b7-97d7-56ef8c51c337" (UID: "ac2760b4-cdc3-42b7-97d7-56ef8c51c337"). InnerVolumeSpecName "kube-api-access-hm4f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:40.873940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.873911    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4f5\" (UniqueName: \"kubernetes.io/projected/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-kube-api-access-hm4f5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:40.873940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:40.873938    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2760b4-cdc3-42b7-97d7-56ef8c51c337-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:41.472288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:41.472244    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn" event={"ID":"ac2760b4-cdc3-42b7-97d7-56ef8c51c337","Type":"ContainerDied","Data":"3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70"}
Oct 30 18:45:41.472569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:41.472550    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a17a2378ad051c8e75740133f1e63c8a2758b94bab141e1ac4f7e7f83d33d70"
Oct 30 18:45:41.472780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:41.472761    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"
Oct 30 18:45:51.631500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.631457    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:45:51.632005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.631993    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ac2760b4-cdc3-42b7-97d7-56ef8c51c337" containerName="revoke-keys"
Oct 30 18:45:51.632005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.632006    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2760b4-cdc3-42b7-97d7-56ef8c51c337" containerName="revoke-keys"
Oct 30 18:45:51.632184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.632111    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2760b4-cdc3-42b7-97d7-56ef8c51c337" containerName="revoke-keys"
Oct 30 18:45:51.635620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.635593    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.661677 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.661655    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:45:51.661848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.661769    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.661986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.661970    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662149    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662246    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662350 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662288    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662350 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662322    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662381    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662474    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8m4w\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.662608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.662588    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763415    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763452    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763469    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763491    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763530    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763579    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8m4w\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763626    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763742    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763773    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.763983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.763880    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.764173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.764153    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.764227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.764199    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.764227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.764217    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.764417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.764397    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.766126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.766101    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.766224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.766150    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.820101 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.820049    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.820284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.820182    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8m4w\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\") pod \"maas-default-gateway-openshift-default-855f9574cf-wd8zw\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:51.949875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:51.949807    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:52.260477 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.260450    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:45:52.262496 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:45:52.262467    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c896eb_26e8_4242_b81c_52336f28e3a8.slice/crio-240bd0b38f22e35372af8775bf1c7ead9493c35d771cac2bca1b78aa79ac4aa7 WatchSource:0}: Error finding container 240bd0b38f22e35372af8775bf1c7ead9493c35d771cac2bca1b78aa79ac4aa7: Status 404 returned error can't find the container with id 240bd0b38f22e35372af8775bf1c7ead9493c35d771cac2bca1b78aa79ac4aa7
Oct 30 18:45:52.265253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.265223    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:45:52.265317 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.265291    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:45:52.265359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.265320    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:45:52.574572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.574487    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" event={"ID":"41c896eb-26e8-4242-b81c-52336f28e3a8","Type":"ContainerStarted","Data":"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"}
Oct 30 18:45:52.574572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.574560    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" event={"ID":"41c896eb-26e8-4242-b81c-52336f28e3a8","Type":"ContainerStarted","Data":"240bd0b38f22e35372af8775bf1c7ead9493c35d771cac2bca1b78aa79ac4aa7"}
Oct 30 18:45:52.611125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.610674    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podStartSLOduration=1.610649071 podStartE2EDuration="1.610649071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:52.605737207 +0000 UTC m=+772.869202156" watchObservedRunningTime="2026-10-30 18:45:52.610649071 +0000 UTC m=+772.874114031"
Oct 30 18:45:52.950068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:52.949989    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:45:53.059557 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:45:53.059523    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:45:53.952935 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:53.952903    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:53.953347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:53.952965    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:54.950811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:54.950643    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:54.950811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:54.950722    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:55.950950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:55.950914    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:55.951371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:55.950982    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:56.951071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:56.951036    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:56.951521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:56.951129    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:57.950806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:57.950775    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:57.950973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:57.950829    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:58.951155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:58.951119    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:58.951644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:58.951185    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:59.651641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.651600    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 18:45:59.656260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.656240    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.684819 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.684795    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 18:45:59.784259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784220    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784411 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784281    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784407    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784456    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784555    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784602    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784669    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784705    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px574\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.784852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.784722    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.886281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.886254    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887550    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887739    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887803    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px574\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887829    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887880    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.887975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887899    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887973    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.888025    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.888058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.888218    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.887250    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.888419    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.888683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.888666    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.890901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.890880    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.891608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.891585    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.902309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.902235    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.902464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.902442    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px574\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:45:59.951001 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.950973    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 18:45:59.951137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.951029    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 18:45:59.972476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:45:59.972447    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:46:00.286805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.286685    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 18:46:00.292358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.292312    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:00.292497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.292477    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:00.292560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.292535    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:00.666109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.665308    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" event={"ID":"f163f3ff-6512-4081-8e1b-17f8bb6ddf55","Type":"ContainerStarted","Data":"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"}
Oct 30 18:46:00.666109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.665387    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" event={"ID":"f163f3ff-6512-4081-8e1b-17f8bb6ddf55","Type":"ContainerStarted","Data":"1db0ba5f6dab37ad6c7d40c0e9404482b4764ee71901b7fdb90df25230e526d2"}
Oct 30 18:46:00.694897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.694822    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podStartSLOduration=1.694805409 podStartE2EDuration="1.694805409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:00.693398464 +0000 UTC m=+780.956863435" watchObservedRunningTime="2026-10-30 18:46:00.694805409 +0000 UTC m=+780.958270357"
Oct 30 18:46:00.974168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:00.973207    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:46:01.038904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:01.038862    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:46:01.040296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:01.040196    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:46:01.041390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:01.041366    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:46:01.974517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:01.974483    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:01.974988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:01.974563    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:02.978110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:02.977191    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:02.978110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:02.977251    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:03.815114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:03.814164    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:03.950112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:03.945163    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:03.950112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:03.945487    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 18:46:03.985741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:03.975891    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:03.985741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:03.975938    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:04.408112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.407181    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:04.493114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.492267    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:04.668291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.668183    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:04.950170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.947358    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:04.950170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.947741    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:46:04.991144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.987958    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:04.991144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:04.988007    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:05.167116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.126846    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:05.408317 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.408253    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:05.432575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.432537    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:46:05.495114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.494157    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:05.495114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.494850    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:46:05.670185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.669588    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:05.670185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.669984    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:46:05.955185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.952375    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:05.983398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.983369    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:05.983656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:05.983632    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:06.268881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.268853    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:46:06.417456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.417412    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:06.499111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.496155    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:46:06.947111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.943258    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mz99s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:46:06.947111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.943330    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:06.963108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.955159    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:06.981111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.978512    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:06.981111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:06.978557    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:07.281110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:07.271173    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:07.419115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:07.418196    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:07.500111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:07.499179    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:07.677390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:07.677309    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:46:07.789786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:07.789756    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:46:08.106112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:08.092440    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:46:08.106112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:08.092491    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:46:08.291017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:08.290988    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:46:09.971281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:09.971251    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:46:09.971747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:09.971731    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:46:09.979969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:09.979934    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 18:46:14.391507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.391474    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 18:46:14.395893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.395869    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.407522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.407500    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"distinct-2-w4-da70a5-kserve-self-signed-certs\""
Oct 30 18:46:14.411302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.411277    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 18:46:14.502499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502472    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.502647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.502717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502690    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.502792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502771    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.502846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502813    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.502957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.502938    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqjh\" (UniqueName: \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604582    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqjh\" (UniqueName: \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604637    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604689    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604790    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604851    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.604966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.604881    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.605445 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.605417    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.605687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.605659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.605893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.605725    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.607716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.607688    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.608938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.608911    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.619900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.619878    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqjh\" (UniqueName: \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\") pod \"distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.711037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.710968    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:14.995410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:14.995372    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 18:46:14.996895 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:14.996865    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4f78b43_62f9_4fee_851a_8bf6d8cf70e5.slice/crio-54fb958065bd1f8416f33681041a4c8b9cdfc0b2f1296fd2138754df7756af11 WatchSource:0}: Error finding container 54fb958065bd1f8416f33681041a4c8b9cdfc0b2f1296fd2138754df7756af11: Status 404 returned error can't find the container with id 54fb958065bd1f8416f33681041a4c8b9cdfc0b2f1296fd2138754df7756af11
Oct 30 18:46:15.176003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:15.175973    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" event={"ID":"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5","Type":"ContainerStarted","Data":"5ce40812f6ee2ae0251b5153ada52b6bbb80e740b71622c8910df75ac2398673"}
Oct 30 18:46:15.176003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:15.176009    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" event={"ID":"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5","Type":"ContainerStarted","Data":"54fb958065bd1f8416f33681041a4c8b9cdfc0b2f1296fd2138754df7756af11"}
Oct 30 18:46:15.198126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:15.198068    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podStartSLOduration=1.198053749 podStartE2EDuration="1.198053749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:15.19736296 +0000 UTC m=+795.460827953" watchObservedRunningTime="2026-10-30 18:46:15.198053749 +0000 UTC m=+795.461518697"
Oct 30 18:46:15.711339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:15.711308    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:23.066637 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:46:23.066594    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:46:24.711452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:24.711417    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:24.724249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:24.724225    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:25.272605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:25.272577    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:46:26.458859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.458819    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 18:46:26.463885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.463853    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 18:46:26.464031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.463863    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.472698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.471563    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"default-dockercfg-ttpl9\""
Oct 30 18:46:26.472698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.471563    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"kube-root-ca.crt\""
Oct 30 18:46:26.472698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.471613    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"distinct-w2-8a92cc-kserve-self-signed-certs\""
Oct 30 18:46:26.472698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.471638    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"openshift-service-ca.crt\""
Oct 30 18:46:26.564339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564308    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.564518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564427    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.564518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564473    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.564612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564527    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hrc\" (UniqueName: \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.564612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564556    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.564690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.564643    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666242    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666289    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666329    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hrc\" (UniqueName: \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666349    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666426    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666513    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.666828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666804    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.667013 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666888    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.667131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.666951    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.668947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.668921    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.669278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.669260    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.680181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.680133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hrc\" (UniqueName: \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\") pod \"distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:26.782357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:26.782325    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:27.073060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.072975    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 18:46:27.078723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.078701    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.083240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.083219    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 18:46:27.083399 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:27.083237    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c74c55_4e95_4d65_bd8d_b36572307f6c.slice/crio-d9528f4bea4189353015574b95c7b6be8d5421643e186e3736a9dad30572b954 WatchSource:0}: Error finding container d9528f4bea4189353015574b95c7b6be8d5421643e186e3736a9dad30572b954: Status 404 returned error can't find the container with id d9528f4bea4189353015574b95c7b6be8d5421643e186e3736a9dad30572b954
Oct 30 18:46:27.101817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.101741    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 18:46:27.174557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174531    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174571    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174700    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174737    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174821    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smn7z\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174854    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174870    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174905    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.174996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.174983    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.276723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276688    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.276906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276738    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.276906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276824    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.276906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276854    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276909    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smn7z\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276939    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.276962    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.277004    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.277060    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.277847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.277824    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.282796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.278512    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.282796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.278977    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.282796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.279022    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.282796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.281391    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.282796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.281673    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.283298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.283274    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.285857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.285828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" event={"ID":"37c74c55-4e95-4d65-bd8d-b36572307f6c","Type":"ContainerStarted","Data":"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"}
Oct 30 18:46:27.285984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.285885    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" event={"ID":"37c74c55-4e95-4d65-bd8d-b36572307f6c","Type":"ContainerStarted","Data":"d9528f4bea4189353015574b95c7b6be8d5421643e186e3736a9dad30572b954"}
Oct 30 18:46:27.292899 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.292874    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.293059 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.293043    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smn7z\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.314465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.314418    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podStartSLOduration=1.314402999 podStartE2EDuration="1.314402999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:27.309521328 +0000 UTC m=+807.572986314" watchObservedRunningTime="2026-10-30 18:46:27.314402999 +0000 UTC m=+807.577867952"
Oct 30 18:46:27.415604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.415536    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:27.715735 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:27.715704    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88fb979_a4e3_4b70_92eb_a2030d0540f6.slice/crio-ecd0c156fb3113f840c8d10b31178d97f9a3c9c9bd08ac431ff431f2c241893d WatchSource:0}: Error finding container ecd0c156fb3113f840c8d10b31178d97f9a3c9c9bd08ac431ff431f2c241893d: Status 404 returned error can't find the container with id ecd0c156fb3113f840c8d10b31178d97f9a3c9c9bd08ac431ff431f2c241893d
Oct 30 18:46:27.718836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.718804    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:27.718947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.718871    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:27.718947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.718900    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:27.719181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.719160    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 18:46:27.782850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:27.782824    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:28.297871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.297839    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" event={"ID":"c88fb979-a4e3-4b70-92eb-a2030d0540f6","Type":"ContainerStarted","Data":"37a46763469e5239bab9688fe59d8bafc109a599d416c093966f315cdf93ea32"}
Oct 30 18:46:28.298046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.297899    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" event={"ID":"c88fb979-a4e3-4b70-92eb-a2030d0540f6","Type":"ContainerStarted","Data":"ecd0c156fb3113f840c8d10b31178d97f9a3c9c9bd08ac431ff431f2c241893d"}
Oct 30 18:46:28.325423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.325363    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podStartSLOduration=1.325346408 podStartE2EDuration="1.325346408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:28.323410695 +0000 UTC m=+808.586875644" watchObservedRunningTime="2026-10-30 18:46:28.325346408 +0000 UTC m=+808.588811358"
Oct 30 18:46:28.417426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.417387    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:28.420042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.420007    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:28.420244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:28.420063    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:29.415997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:29.415958    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:29.416572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:29.416011    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:30.421110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.416317    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:30.421110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.416375    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:30.912114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.911210    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"]
Oct 30 18:46:30.926344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.926314    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"]
Oct 30 18:46:30.926585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.926423    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.935014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.934541    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 18:46:30.935014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.934621    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 18:46:30.935014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.934674    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 18:46:30.938473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.938454    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7h4l8\""
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.947748    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.947790    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.947840    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.947947    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.948013    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm95p\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-kube-api-access-qm95p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.948050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:30.949859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:30.948126    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.049785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049749    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.049785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049788    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.050193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049831    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.050193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049896    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.050193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049951    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm95p\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-kube-api-access-qm95p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.050193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049976    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.050193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.049996    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.051282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.051251    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.052845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.052821    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.053515 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.053343    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.053515 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.053466    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.053732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.053713    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.065820 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.065044    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm95p\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-kube-api-access-qm95p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.066053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.066035    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q46vl\" (UID: \"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.244790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.244718    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:31.426294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.421546    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:31.426294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.421610    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:31.862985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.862953    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"]
Oct 30 18:46:31.872765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:31.872740    5065 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:46:32.383327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:32.383291    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl" event={"ID":"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3","Type":"ContainerStarted","Data":"bddc118bb85ddfd5a0d4f1f8203f229a1f18b169954730fbeae547816de150b8"}
Oct 30 18:46:32.418113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:32.416147    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:32.418113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:32.416202    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:33.416205 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.416176    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:33.416868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.416750    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:33.847769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.847736    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"]
Oct 30 18:46:33.851866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.851845    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:33.862386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.862366    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:46:33.864793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.864769    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:46:33.870839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.870592    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"]
Oct 30 18:46:33.949840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.949801    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwhj\" (UniqueName: \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:33.950027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.949872    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:33.950027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:33.949939    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.051173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.051137    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.051348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.051186    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.051348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.051338    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwhj\" (UniqueName: \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.051869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.051837    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.054816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.054792    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.065532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.065505    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwhj\" (UniqueName: \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\") pod \"maas-api-revoke-sub-6ffc6e92-h4t4g\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.180615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.180541    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:34.416987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.416882    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:46:34.416987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.416940    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:46:34.493880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:34.493797    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"]
Oct 30 18:46:34.499245 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:34.499208    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fa1054_064e_441a_9e61_d0716bc62a07.slice/crio-93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce WatchSource:0}: Error finding container 93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce: Status 404 returned error can't find the container with id 93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce
Oct 30 18:46:35.436119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:35.435336    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g" event={"ID":"a4fa1054-064e-441a-9e61-d0716bc62a07","Type":"ContainerStarted","Data":"924bcf52fcb4b2c82473a695d7f43b98a088e0878dc43794b85c1ea3a552cf28"}
Oct 30 18:46:35.436119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:35.435408    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g" event={"ID":"a4fa1054-064e-441a-9e61-d0716bc62a07","Type":"ContainerStarted","Data":"93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce"}
Oct 30 18:46:35.460848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:35.460797    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g" podStartSLOduration=2.460784098 podStartE2EDuration="2.460784098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:35.455725756 +0000 UTC m=+815.719190722" watchObservedRunningTime="2026-10-30 18:46:35.460784098 +0000 UTC m=+815.724249045"
Oct 30 18:46:36.448126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.448069    5065 generic.go:356] "Generic (PLEG): container finished" podID="a4fa1054-064e-441a-9e61-d0716bc62a07" containerID="924bcf52fcb4b2c82473a695d7f43b98a088e0878dc43794b85c1ea3a552cf28" exitCode=0
Oct 30 18:46:36.448528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.448135    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g" event={"ID":"a4fa1054-064e-441a-9e61-d0716bc62a07","Type":"ContainerDied","Data":"924bcf52fcb4b2c82473a695d7f43b98a088e0878dc43794b85c1ea3a552cf28"}
Oct 30 18:46:36.458273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.458230    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:46:36.458425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.458283    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:36.467814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.467785    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:36.467945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.467878    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:36.645791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.645759    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:36.646255 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.646231    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:36.648079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.648059    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 18:46:36.783241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.783214    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:36.795790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:36.795767    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:37.458846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.458808    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl" event={"ID":"1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3","Type":"ContainerStarted","Data":"b2fa74c23e7c3e4945d31bd2c0d09d2f640f71373e945142cfd84f38c27eb604"}
Oct 30 18:46:37.460726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.460704    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:37.463072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.463051    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl"
Oct 30 18:46:37.475644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.475613    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:46:37.492263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.492202    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q46vl" podStartSLOduration=2.898501689 podStartE2EDuration="7.492183787s" totalImagesPullingTime="4.593682098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:37.488608164 +0000 UTC m=+817.752073114" watchObservedRunningTime="2026-10-30 18:46:37.492183787 +0000 UTC m=+817.755648735"
Oct 30 18:46:37.675480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.675454    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:37.830573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.830542    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\") pod \"a4fa1054-064e-441a-9e61-d0716bc62a07\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") "
Oct 30 18:46:37.830773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.830664    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\") pod \"a4fa1054-064e-441a-9e61-d0716bc62a07\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") "
Oct 30 18:46:37.830773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.830693    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\" (UniqueName: \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\") pod \"a4fa1054-064e-441a-9e61-d0716bc62a07\" (UID: \"a4fa1054-064e-441a-9e61-d0716bc62a07\") "
Oct 30 18:46:37.830883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.830793    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca" pod "a4fa1054-064e-441a-9e61-d0716bc62a07" (UID: "a4fa1054-064e-441a-9e61-d0716bc62a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:37.831219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.831200    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:37.832890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.832865    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca" pod "a4fa1054-064e-441a-9e61-d0716bc62a07" (UID: "a4fa1054-064e-441a-9e61-d0716bc62a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:46:37.832971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.832956    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj" pod "a4fa1054-064e-441a-9e61-d0716bc62a07" (UID: "a4fa1054-064e-441a-9e61-d0716bc62a07"). InnerVolumeSpecName "kube-api-access-4mwhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:37.931817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.931792    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa1054-064e-441a-9e61-d0716bc62a07-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:37.931817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:37.931817    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwhj\" (UniqueName: \"kubernetes.io/projected/a4fa1054-064e-441a-9e61-d0716bc62a07-kube-api-access-4mwhj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:38.019228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.019195    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"]
Oct 30 18:46:38.019873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.019853    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a4fa1054-064e-441a-9e61-d0716bc62a07" containerName="revoke-keys"
Oct 30 18:46:38.019982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.019875    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fa1054-064e-441a-9e61-d0716bc62a07" containerName="revoke-keys"
Oct 30 18:46:38.020044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.020002    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fa1054-064e-441a-9e61-d0716bc62a07" containerName="revoke-keys"
Oct 30 18:46:38.023813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.023795    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.039368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.039345    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"]
Oct 30 18:46:38.136450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.136364    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.136450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.136414    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p55p\" (UniqueName: \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.136642 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.136497    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.237875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.237836    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.238031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.237901    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p55p\" (UniqueName: \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.238031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.237938    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.238517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.238496    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.240559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.240537    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.268514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.268488    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p55p\" (UniqueName: \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\") pod \"maas-api-revoke-sub-d7c5fae5-nbx9r\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.341548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.341522    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:38.472319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.472284    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"
Oct 30 18:46:38.473640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.472825    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g" event={"ID":"a4fa1054-064e-441a-9e61-d0716bc62a07","Type":"ContainerDied","Data":"93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce"}
Oct 30 18:46:38.473640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.472871    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93463cbc3bcf10245d63a2f1c76449290f95a566d76269b291d68f08314ad0ce"
Oct 30 18:46:38.623886 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:38.623849    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda014dfb8_1c55_4b2e_8348_b8cc4daf4bd7.slice/crio-35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9 WatchSource:0}: Error finding container 35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9: Status 404 returned error can't find the container with id 35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9
Oct 30 18:46:38.625593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:38.625476    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"]
Oct 30 18:46:39.481655 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:39.481622    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r" event={"ID":"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7","Type":"ContainerStarted","Data":"6194fadb229c0d869a6344102383e87fe04053703d03dc8ace69e53446ec8f2f"}
Oct 30 18:46:39.482040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:39.481680    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r" event={"ID":"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7","Type":"ContainerStarted","Data":"35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9"}
Oct 30 18:46:39.504540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:39.504497    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r" podStartSLOduration=2.504484228 podStartE2EDuration="2.504484228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:39.502283375 +0000 UTC m=+819.765748343" watchObservedRunningTime="2026-10-30 18:46:39.504484228 +0000 UTC m=+819.767949178"
Oct 30 18:46:40.492697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:40.492663    5065 generic.go:356] "Generic (PLEG): container finished" podID="a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" containerID="6194fadb229c0d869a6344102383e87fe04053703d03dc8ace69e53446ec8f2f" exitCode=0
Oct 30 18:46:40.493079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:40.492698    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r" event={"ID":"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7","Type":"ContainerDied","Data":"6194fadb229c0d869a6344102383e87fe04053703d03dc8ace69e53446ec8f2f"}
Oct 30 18:46:41.712258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.712228    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:41.897698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.897643    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\") pod \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") "
Oct 30 18:46:41.897910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.897825    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\" (UniqueName: \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\") pod \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") "
Oct 30 18:46:41.897910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.897899    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\") pod \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\" (UID: \"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7\") "
Oct 30 18:46:41.898636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.898598    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca" pod "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" (UID: "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:41.900499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.900474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca" pod "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" (UID: "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:46:41.902436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.902407    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p" pod "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" (UID: "a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7"). InnerVolumeSpecName "kube-api-access-9p55p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:41.999989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.999945    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:42.000187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:41.999996    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p55p\" (UniqueName: \"kubernetes.io/projected/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-kube-api-access-9p55p\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:42.000187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:42.000023    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:42.516004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:42.515966    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r" event={"ID":"a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7","Type":"ContainerDied","Data":"35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9"}
Oct 30 18:46:42.516213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:42.516024    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35fe184df92eb955997726443a8925b982d3c0c39d475a2b518c2ca6695363b9"
Oct 30 18:46:42.516213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:42.515977    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"
Oct 30 18:46:45.098130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.098064    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 18:46:45.099737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.099706    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" containerName="revoke-keys"
Oct 30 18:46:45.099924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.099908    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" containerName="revoke-keys"
Oct 30 18:46:45.100241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.100222    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" containerName="revoke-keys"
Oct 30 18:46:45.181981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.181945    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 18:46:45.182248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.182223    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.279045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279006    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.279412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279387    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.279591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279573    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.279755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279738    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.279885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279868    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.280006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.279990    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5wt\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.306339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.306306    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.306629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.306606    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.306805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.306787    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507046    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507197    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507250    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507292    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507338    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507367    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507397    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5wt\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507461    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.509221 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.507518    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.523133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.526327    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.536751    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.541227    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.554992    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.564460    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.578164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.570879    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5wt\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.675112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.656783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.675112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.657017    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:45.813418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:45.813320    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:46.506394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.505171    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:46.533112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.531076    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"]
Oct 30 18:46:46.670411 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.670336    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.729128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.684318    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:46:46.729128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.688527    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:46:46.759869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.759827    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj46b\" (UniqueName: \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.760035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.759903    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.760199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.760059    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.770891 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.770701    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"]
Oct 30 18:46:46.829568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.829535    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:46.830705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.830679    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:46.830904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.830884    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:46:46.840941 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.840383    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 18:46:46.862351 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.861884    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj46b\" (UniqueName: \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.862351 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.861943    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.862351 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.862111    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.863644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.863199    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.869724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.869697    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:46.876538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:46.876512    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj46b\" (UniqueName: \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\") pod \"maas-api-revoke-sub-a8d0d074-9gzm5\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:47.114535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.114267    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:47.480627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.480500    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"]
Oct 30 18:46:47.775795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.775752    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5" event={"ID":"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d","Type":"ContainerStarted","Data":"8553c4f232b6f6952603409256a2c89ba2c25b89da098a1bc272b7c8744550a1"}
Oct 30 18:46:47.777578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.777552    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" event={"ID":"19a6e8be-e060-4dab-9347-466160f4c3b8","Type":"ContainerStarted","Data":"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"}
Oct 30 18:46:47.777719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.777609    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" event={"ID":"19a6e8be-e060-4dab-9347-466160f4c3b8","Type":"ContainerStarted","Data":"6e3edb995798f5c94dd1ded8b84f4887b30430c0d3983fd2d6586a7f97d90967"}
Oct 30 18:46:47.817996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.817926    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podStartSLOduration=3.8179064240000002 podStartE2EDuration="3.817906424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:47.812643218 +0000 UTC m=+828.076108195" watchObservedRunningTime="2026-10-30 18:46:47.817906424 +0000 UTC m=+828.081371373"
Oct 30 18:46:47.818417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.818396    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:47.820747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.820442    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:47.820747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:47.820490    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:48.814588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:48.814541    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:48.815263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:48.814603    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:48.818664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:48.818637    5065 generic.go:356] "Generic (PLEG): container finished" podID="d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" containerID="3ca0372ae2dd62d33914b058887d819b06f9603ac3b3e241466edeee4ebf09f7" exitCode=0
Oct 30 18:46:48.818848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:48.818680    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5" event={"ID":"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d","Type":"ContainerDied","Data":"3ca0372ae2dd62d33914b058887d819b06f9603ac3b3e241466edeee4ebf09f7"}
Oct 30 18:46:49.815530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:49.815495    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:49.816141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:49.816107    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:50.111113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.101459    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:50.138966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.138935    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\") pod \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") "
Oct 30 18:46:50.139277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.139256    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\" (UniqueName: \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\") pod \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") "
Oct 30 18:46:50.143498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.143464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca" pod "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" (UID: "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:46:50.144124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.144071    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b" pod "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" (UID: "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d"). InnerVolumeSpecName "kube-api-access-lj46b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:50.242293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.242252    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\") pod \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\" (UID: \"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d\") "
Oct 30 18:46:50.242988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.242705    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:50.242988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.242733    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj46b\" (UniqueName: \"kubernetes.io/projected/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-kube-api-access-lj46b\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:50.244926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.243017    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca" pod "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" (UID: "d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:50.344458 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.344376    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:50.498113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.494984    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"]
Oct 30 18:46:50.498113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.495917    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" containerName="revoke-keys"
Oct 30 18:46:50.498113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.495936    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" containerName="revoke-keys"
Oct 30 18:46:50.498113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.496121    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" containerName="revoke-keys"
Oct 30 18:46:50.522356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.522318    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"]
Oct 30 18:46:50.522533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.522440    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.548117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.546912    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.548117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.546994    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tp2\" (UniqueName: \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.548117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.547171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.648563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.648475    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.648707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.648618    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.648707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.648694    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tp2\" (UniqueName: \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.649282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.649255    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.651212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.651192    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.663602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.663571    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tp2\" (UniqueName: \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\") pod \"maas-api-revoke-sub-b702625a-2gwp2\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:50.814796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.814753    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:50.814993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.814834    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:50.844267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.844233    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5" event={"ID":"d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d","Type":"ContainerDied","Data":"8553c4f232b6f6952603409256a2c89ba2c25b89da098a1bc272b7c8744550a1"}
Oct 30 18:46:50.844635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.844274    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"
Oct 30 18:46:50.844635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.844292    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8553c4f232b6f6952603409256a2c89ba2c25b89da098a1bc272b7c8744550a1"
Oct 30 18:46:50.846169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:50.846149    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:51.166981 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:46:51.166949    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecac9af_05e0_43c7_b651_6cd051eccb42.slice/crio-0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a WatchSource:0}: Error finding container 0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a: Status 404 returned error can't find the container with id 0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a
Oct 30 18:46:51.168301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.168276    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"]
Oct 30 18:46:51.814336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.814295    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:51.814511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.814356    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:51.853380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.853347    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2" event={"ID":"3ecac9af-05e0-43c7-b651-6cd051eccb42","Type":"ContainerStarted","Data":"751e21d5d8060b000246bb4435c86e29311f64a91ff5d7889ad5739bb8c6a8de"}
Oct 30 18:46:51.853894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.853406    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2" event={"ID":"3ecac9af-05e0-43c7-b651-6cd051eccb42","Type":"ContainerStarted","Data":"0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a"}
Oct 30 18:46:51.876606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:51.876501    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2" podStartSLOduration=1.8764827149999999 podStartE2EDuration="1.876482715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:51.87590077 +0000 UTC m=+832.139365718" watchObservedRunningTime="2026-10-30 18:46:51.876482715 +0000 UTC m=+832.139947664"
Oct 30 18:46:52.818111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:52.814222    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:52.818111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:52.814294    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:52.862056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:52.862022    5065 generic.go:356] "Generic (PLEG): container finished" podID="3ecac9af-05e0-43c7-b651-6cd051eccb42" containerID="751e21d5d8060b000246bb4435c86e29311f64a91ff5d7889ad5739bb8c6a8de" exitCode=0
Oct 30 18:46:52.862517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:52.862118    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2" event={"ID":"3ecac9af-05e0-43c7-b651-6cd051eccb42","Type":"ContainerDied","Data":"751e21d5d8060b000246bb4435c86e29311f64a91ff5d7889ad5739bb8c6a8de"}
Oct 30 18:46:53.057974 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:46:53.057940    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:46:53.814588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:53.814557    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 18:46:53.814793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:53.814613    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 18:46:54.054806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.054783    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:54.132194 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.132115    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\") pod \"3ecac9af-05e0-43c7-b651-6cd051eccb42\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") "
Oct 30 18:46:54.132351 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.132271    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\") pod \"3ecac9af-05e0-43c7-b651-6cd051eccb42\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") "
Oct 30 18:46:54.132419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.132351    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\" (UniqueName: \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\") pod \"3ecac9af-05e0-43c7-b651-6cd051eccb42\" (UID: \"3ecac9af-05e0-43c7-b651-6cd051eccb42\") "
Oct 30 18:46:54.132606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.132576    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca" pod "3ecac9af-05e0-43c7-b651-6cd051eccb42" (UID: "3ecac9af-05e0-43c7-b651-6cd051eccb42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:54.133330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.133304    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:54.136314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.136287    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca" pod "3ecac9af-05e0-43c7-b651-6cd051eccb42" (UID: "3ecac9af-05e0-43c7-b651-6cd051eccb42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:46:54.146423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.146399    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2" pod "3ecac9af-05e0-43c7-b651-6cd051eccb42" (UID: "3ecac9af-05e0-43c7-b651-6cd051eccb42"). InnerVolumeSpecName "kube-api-access-68tp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:54.234974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.234939    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecac9af-05e0-43c7-b651-6cd051eccb42-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:54.234974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.234969    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68tp2\" (UniqueName: \"kubernetes.io/projected/3ecac9af-05e0-43c7-b651-6cd051eccb42-kube-api-access-68tp2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:54.818963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.818932    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:54.819345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.819317    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:54.820128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.820104    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 18:46:54.877558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.877518    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"
Oct 30 18:46:54.877845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.877817    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2" event={"ID":"3ecac9af-05e0-43c7-b651-6cd051eccb42","Type":"ContainerDied","Data":"0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a"}
Oct 30 18:46:54.877981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:54.877865    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e60ebc01e26f4503ff98ffb4b91fdfeb5b087da8a70d5fffc4686c2973ad85a"
Oct 30 18:46:58.452113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:58.435984    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:58.470442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:58.470403    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:46:58.990509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:58.985529    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:46:59.137580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.137539    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.355944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.355888    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.431113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.415799    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.440648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.418165    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:46:59.446233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.440169    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:46:59.446692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.446670    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:46:59.472192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.472153    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.499455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:46:59.499428    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:47:00.045112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.013249    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.184110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.180408    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.318021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.317923    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.373575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.373018    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:47:00.408140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.407013    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:47:00.408140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.407154    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:47:00.408140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.407192    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:47:00.474670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.464116    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:47:00.582178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.490389    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.582178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.490913    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:47:00.582178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.518844    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.582178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.544314    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.582178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:00.550847    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:01.129460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.129419    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:01.129746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.129716    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.159237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.159195    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:47:01.235814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.235770    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.340066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.340014    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:01.340457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.340416    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.425216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.418916    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.465576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.465529    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.517263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.517185    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.577108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.573277    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.577108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.574800    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.806119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:01.801334    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:02.199607 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.199507    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:47:02.200624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.200605    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:47:02.228035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.227991    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:02.238243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.238212    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:47:02.265336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.265216    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wd8zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:02.265336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.265271    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:02.299121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.298347    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:47:02.379112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.357219    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:02.379112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.368327    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:47:02.467665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.467404    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:47:02.577713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.572507    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:47:02.753912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:02.753822    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:47:15.582274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:15.582167    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:47:15.713227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:15.713175    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.046208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.046175    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v8w6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:16.046621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.046589    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v8w6k" podUID="ca1f838b-93cf-425f-b9ac-232488fca33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.311042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.311006    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:16.318424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.318389    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.478217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.478174    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:47:16.538229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.538178    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.584274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.584170    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.585562 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:16.585536    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:17.586165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:17.586113    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:18.172863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.171762    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:18.770965 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.770932    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"]
Oct 30 18:47:18.773527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.773478    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3ecac9af-05e0-43c7-b651-6cd051eccb42" containerName="revoke-keys"
Oct 30 18:47:18.773720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.773703    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecac9af-05e0-43c7-b651-6cd051eccb42" containerName="revoke-keys"
Oct 30 18:47:18.832616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.832580    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecac9af-05e0-43c7-b651-6cd051eccb42" containerName="revoke-keys"
Oct 30 18:47:18.842277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.842245    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"]
Oct 30 18:47:18.842592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.842574    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:18.860612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.860591    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:18.860801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:18.860789    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:47:19.008904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.008870    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.009302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.009279    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.009615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.009596    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrkq\" (UniqueName: \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.111252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.111161    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.111415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.111317    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.111491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.111467    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrkq\" (UniqueName: \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.112418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.112388    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.114024 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.114001    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.126319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.126292    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrkq\" (UniqueName: \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\") pod \"maas-api-revoke-sub-fae71fd2-lpvh9\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.182801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.182773    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:19.488597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:19.488569    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"]
Oct 30 18:47:19.490120 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:19.490064    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73096e9_d6ec_4b22_af51_00d1da4110db.slice/crio-e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe WatchSource:0}: Error finding container e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe: Status 404 returned error can't find the container with id e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe
Oct 30 18:47:20.483599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:20.483571    5065 generic.go:356] "Generic (PLEG): container finished" podID="f73096e9-d6ec-4b22-af51-00d1da4110db" containerID="719f900f39e5f49662b26163f4a9350a7e9102f0f7e35cd39be954b5a2e834ed" exitCode=0
Oct 30 18:47:20.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:20.483672    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9" event={"ID":"f73096e9-d6ec-4b22-af51-00d1da4110db","Type":"ContainerDied","Data":"719f900f39e5f49662b26163f4a9350a7e9102f0f7e35cd39be954b5a2e834ed"}
Oct 30 18:47:20.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:20.483738    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9" event={"ID":"f73096e9-d6ec-4b22-af51-00d1da4110db","Type":"ContainerStarted","Data":"e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe"}
Oct 30 18:47:21.647394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.647369    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:21.747299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.747210    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\") pod \"f73096e9-d6ec-4b22-af51-00d1da4110db\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") "
Oct 30 18:47:21.747472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.747314    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\") pod \"f73096e9-d6ec-4b22-af51-00d1da4110db\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") "
Oct 30 18:47:21.747472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.747341    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\" (UniqueName: \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\") pod \"f73096e9-d6ec-4b22-af51-00d1da4110db\" (UID: \"f73096e9-d6ec-4b22-af51-00d1da4110db\") "
Oct 30 18:47:21.747693 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.747668    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca" pod "f73096e9-d6ec-4b22-af51-00d1da4110db" (UID: "f73096e9-d6ec-4b22-af51-00d1da4110db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:21.747856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.747833    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:21.749593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.749567    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq" pod "f73096e9-d6ec-4b22-af51-00d1da4110db" (UID: "f73096e9-d6ec-4b22-af51-00d1da4110db"). InnerVolumeSpecName "kube-api-access-kgrkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:21.749735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.749703    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca" pod "f73096e9-d6ec-4b22-af51-00d1da4110db" (UID: "f73096e9-d6ec-4b22-af51-00d1da4110db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:21.849457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.849421    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgrkq\" (UniqueName: \"kubernetes.io/projected/f73096e9-d6ec-4b22-af51-00d1da4110db-kube-api-access-kgrkq\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:21.849457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:21.849456    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73096e9-d6ec-4b22-af51-00d1da4110db-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:22.498824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:22.498799    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"
Oct 30 18:47:22.498995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:22.498836    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9" event={"ID":"f73096e9-d6ec-4b22-af51-00d1da4110db","Type":"ContainerDied","Data":"e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe"}
Oct 30 18:47:22.498995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:22.498883    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e464839fa6c9d22463c9269088798a45f8b2be871ccd73724d4c9c3692f384fe"
Oct 30 18:47:23.057133 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:47:23.057074    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:47:29.021157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.021116    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"]
Oct 30 18:47:29.022017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.021992    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f73096e9-d6ec-4b22-af51-00d1da4110db" containerName="revoke-keys"
Oct 30 18:47:29.022192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.022019    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73096e9-d6ec-4b22-af51-00d1da4110db" containerName="revoke-keys"
Oct 30 18:47:29.022285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.022268    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73096e9-d6ec-4b22-af51-00d1da4110db" containerName="revoke-keys"
Oct 30 18:47:29.027856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.027621    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.035523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.035490    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"]
Oct 30 18:47:29.040233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.039756    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:47:29.040233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.039826    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:29.140319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.140287    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.140319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.140321    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww59h\" (UniqueName: \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.140586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.140501    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.241727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.241693    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.241727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.241730    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww59h\" (UniqueName: \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.241999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.241826    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.242586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.242559    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.245160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.245133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.261828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.261802    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww59h\" (UniqueName: \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\") pod \"maas-api-revoke-sub-1ca49098-9rwf5\" (UID: \"260a295b-a834-4498-9846-db028879c221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.362894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.362818    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:29.676225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:29.668732    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"]
Oct 30 18:47:30.571319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:30.571282    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5" event={"ID":"260a295b-a834-4498-9846-db028879c221","Type":"ContainerStarted","Data":"18642b955dc67bff3d7864927486d34f6aebb74a5589f1fbc9394bb97ad86c8b"}
Oct 30 18:47:30.571740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:30.571346    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5" event={"ID":"260a295b-a834-4498-9846-db028879c221","Type":"ContainerStarted","Data":"be11a22c8474b3edde7149400dc89af2cb704febfcfe9b309822c86c73c5a8f9"}
Oct 30 18:47:30.594984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:30.594937    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5" podStartSLOduration=2.5949207850000002 podStartE2EDuration="2.594920785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:30.591465492 +0000 UTC m=+870.854930473" watchObservedRunningTime="2026-10-30 18:47:30.594920785 +0000 UTC m=+870.858385732"
Oct 30 18:47:31.340552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.340519    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"]
Oct 30 18:47:31.344939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.344922    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.355627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.355606    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"]
Oct 30 18:47:31.474653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.474621    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.474810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.474692    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9tg\" (UniqueName: \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.474885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.474812    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.575812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.575782    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.576318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.575850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9tg\" (UniqueName: \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.576318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.575956    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.576987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.576960    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.578869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.578846    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.581656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.581620    5065 generic.go:356] "Generic (PLEG): container finished" podID="260a295b-a834-4498-9846-db028879c221" containerID="18642b955dc67bff3d7864927486d34f6aebb74a5589f1fbc9394bb97ad86c8b" exitCode=0
Oct 30 18:47:31.581769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.581682    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5" event={"ID":"260a295b-a834-4498-9846-db028879c221","Type":"ContainerDied","Data":"18642b955dc67bff3d7864927486d34f6aebb74a5589f1fbc9394bb97ad86c8b"}
Oct 30 18:47:31.589246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.589224    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9tg\" (UniqueName: \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\") pod \"maas-api-revoke-sub-4d440be8-rx6kn\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.662672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.662594    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:31.957324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:31.957303    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"]
Oct 30 18:47:31.959080 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:31.959049    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa761428_0a3c_41a3_8ea0_6a084ad3d07f.slice/crio-768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596 WatchSource:0}: Error finding container 768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596: Status 404 returned error can't find the container with id 768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596
Oct 30 18:47:32.242439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.242361    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 18:47:32.248021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.248000    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.314128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.310050    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 18:47:32.319825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.319798    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 18:47:32.328655 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.328628    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.353037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.350022    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 18:47:32.388146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388109    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388205    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388269    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388396    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388465    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388544    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388573    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388590    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.388683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.388610    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbzr\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.489478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489441    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.489645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489493    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.489782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489674    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.489782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489729    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksk9\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.489901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489814    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.489901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489871    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.489901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489891    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489908    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489913    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.490031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489927    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.490031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.489930    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.490233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490147    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490197    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490332    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.490413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490402    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490477 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490461    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.490535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490513    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490595    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490617    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490639    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490671    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbzr\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.490785    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.491517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.491489    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.492840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.492738    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.492910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.492867    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.502922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.502898    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.503725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.503701    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbzr\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.563557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.563539    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591761    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591806    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591833    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591886    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591931    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.591989    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592040    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592249    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592302    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksk9\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.592856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592702    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.593012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592854    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.593012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592872    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.593012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.592963    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.595661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.595019    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.595965 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.595943    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.596177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.596158    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.605005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.604972    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn" event={"ID":"fa761428-0a3c-41a3-8ea0-6a084ad3d07f","Type":"ContainerStarted","Data":"c3478bdaa5f842103768e837e885ac070c3d1e89b20ac076bd32966d651d1ca4"}
Oct 30 18:47:32.605252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.605030    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn" event={"ID":"fa761428-0a3c-41a3-8ea0-6a084ad3d07f","Type":"ContainerStarted","Data":"768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596"}
Oct 30 18:47:32.606749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.606624    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.607414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.607393    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksk9\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.635720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.635160    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn" podStartSLOduration=1.635137844 podStartE2EDuration="1.635137844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:32.628034075 +0000 UTC m=+872.891499066" watchObservedRunningTime="2026-10-30 18:47:32.635137844 +0000 UTC m=+872.898602800"
Oct 30 18:47:32.655808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.654728    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:32.832959 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.832929    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:32.877743 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:32.877711    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66d2d86_ee6a_46cd_a891_942f083d3f42.slice/crio-2025ee49ea6cf541fcd8e54be99b47b7f014a96ad996430de5265107def9f9b6 WatchSource:0}: Error finding container 2025ee49ea6cf541fcd8e54be99b47b7f014a96ad996430de5265107def9f9b6: Status 404 returned error can't find the container with id 2025ee49ea6cf541fcd8e54be99b47b7f014a96ad996430de5265107def9f9b6
Oct 30 18:47:32.878345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.878320    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 18:47:32.881539 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.881500    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.881629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.881585    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.881678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.881632    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.967980 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:32.967943    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2976d237_7f8d_48ae_9141_fbaadb689aa6.slice/crio-da70be5678a4148f9926118ec187294797678596725a8de33785d938617b8752 WatchSource:0}: Error finding container da70be5678a4148f9926118ec187294797678596725a8de33785d938617b8752: Status 404 returned error can't find the container with id da70be5678a4148f9926118ec187294797678596725a8de33785d938617b8752
Oct 30 18:47:32.971111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.971055    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.971198 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.971163    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.971237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.971206    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:32.973653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.973630    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 18:47:32.997580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.997437    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\") pod \"260a295b-a834-4498-9846-db028879c221\" (UID: \"260a295b-a834-4498-9846-db028879c221\") "
Oct 30 18:47:32.997685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.997643    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\") pod \"260a295b-a834-4498-9846-db028879c221\" (UID: \"260a295b-a834-4498-9846-db028879c221\") "
Oct 30 18:47:32.997744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.997730    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\" (UniqueName: \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\") pod \"260a295b-a834-4498-9846-db028879c221\" (UID: \"260a295b-a834-4498-9846-db028879c221\") "
Oct 30 18:47:32.999369 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:32.999315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca" pod "260a295b-a834-4498-9846-db028879c221" (UID: "260a295b-a834-4498-9846-db028879c221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:33.000475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.000449    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h" pod "260a295b-a834-4498-9846-db028879c221" (UID: "260a295b-a834-4498-9846-db028879c221"). InnerVolumeSpecName "kube-api-access-ww59h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:33.002432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.002381    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca" pod "260a295b-a834-4498-9846-db028879c221" (UID: "260a295b-a834-4498-9846-db028879c221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:33.099915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.099885    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww59h\" (UniqueName: \"kubernetes.io/projected/260a295b-a834-4498-9846-db028879c221-kube-api-access-ww59h\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:33.099915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.099918    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a295b-a834-4498-9846-db028879c221-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:33.100153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.099936    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a295b-a834-4498-9846-db028879c221-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:33.641868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.641831    5065 generic.go:356] "Generic (PLEG): container finished" podID="fa761428-0a3c-41a3-8ea0-6a084ad3d07f" containerID="c3478bdaa5f842103768e837e885ac070c3d1e89b20ac076bd32966d651d1ca4" exitCode=0
Oct 30 18:47:33.642626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.642597    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn" event={"ID":"fa761428-0a3c-41a3-8ea0-6a084ad3d07f","Type":"ContainerDied","Data":"c3478bdaa5f842103768e837e885ac070c3d1e89b20ac076bd32966d651d1ca4"}
Oct 30 18:47:33.645618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.645576    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5" event={"ID":"260a295b-a834-4498-9846-db028879c221","Type":"ContainerDied","Data":"be11a22c8474b3edde7149400dc89af2cb704febfcfe9b309822c86c73c5a8f9"}
Oct 30 18:47:33.645787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.645771    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be11a22c8474b3edde7149400dc89af2cb704febfcfe9b309822c86c73c5a8f9"
Oct 30 18:47:33.646015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.646001    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"
Oct 30 18:47:33.649939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.649912    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" event={"ID":"2976d237-7f8d-48ae-9141-fbaadb689aa6","Type":"ContainerStarted","Data":"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"}
Oct 30 18:47:33.650851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.650829    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" event={"ID":"2976d237-7f8d-48ae-9141-fbaadb689aa6","Type":"ContainerStarted","Data":"da70be5678a4148f9926118ec187294797678596725a8de33785d938617b8752"}
Oct 30 18:47:33.653046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.652983    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" event={"ID":"b66d2d86-ee6a-46cd-a891-942f083d3f42","Type":"ContainerStarted","Data":"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"}
Oct 30 18:47:33.653046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.653023    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" event={"ID":"b66d2d86-ee6a-46cd-a891-942f083d3f42","Type":"ContainerStarted","Data":"2025ee49ea6cf541fcd8e54be99b47b7f014a96ad996430de5265107def9f9b6"}
Oct 30 18:47:33.655058 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.655042    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:33.732108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.731807    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podStartSLOduration=1.731788516 podStartE2EDuration="1.731788516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:33.723207748 +0000 UTC m=+873.986672697" watchObservedRunningTime="2026-10-30 18:47:33.731788516 +0000 UTC m=+873.995253466"
Oct 30 18:47:33.761653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:33.761592    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podStartSLOduration=1.761574311 podStartE2EDuration="1.761574311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:33.756715603 +0000 UTC m=+874.020180567" watchObservedRunningTime="2026-10-30 18:47:33.761574311 +0000 UTC m=+874.025039259"
Oct 30 18:47:34.564691 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:34.564656    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:34.566328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:34.566298    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:34.566414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:34.566377    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:34.657122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:34.657064    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:34.657616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:34.657135    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:35.077429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.077404    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:35.203142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.192508    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\") pod \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") "
Oct 30 18:47:35.203142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.192603    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\" (UniqueName: \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\") pod \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") "
Oct 30 18:47:35.203142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.192677    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\") pod \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\" (UID: \"fa761428-0a3c-41a3-8ea0-6a084ad3d07f\") "
Oct 30 18:47:35.273263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.273223    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca" pod "fa761428-0a3c-41a3-8ea0-6a084ad3d07f" (UID: "fa761428-0a3c-41a3-8ea0-6a084ad3d07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:35.276277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.276251    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca" pod "fa761428-0a3c-41a3-8ea0-6a084ad3d07f" (UID: "fa761428-0a3c-41a3-8ea0-6a084ad3d07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:35.276431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.276252    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg" pod "fa761428-0a3c-41a3-8ea0-6a084ad3d07f" (UID: "fa761428-0a3c-41a3-8ea0-6a084ad3d07f"). InnerVolumeSpecName "kube-api-access-sg9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:35.295064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.295027    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:35.324763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.324739    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:35.324938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.324923    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9tg\" (UniqueName: \"kubernetes.io/projected/fa761428-0a3c-41a3-8ea0-6a084ad3d07f-kube-api-access-sg9tg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:35.564054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.563973    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:35.564054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.564022    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:35.655987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.655929    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:35.656211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.656037    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:35.673813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.673752    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn" event={"ID":"fa761428-0a3c-41a3-8ea0-6a084ad3d07f","Type":"ContainerDied","Data":"768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596"}
Oct 30 18:47:35.673813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.673779    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"
Oct 30 18:47:35.673813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:35.673804    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="768b283bf62147d9358dd5202dd7014e66a3d16f57bd761134811d3bf4b00596"
Oct 30 18:47:36.565131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:36.564526    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:36.565131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:36.564586    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:36.656139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:36.656081    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:36.656305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:36.656146    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.415568    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"]
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420642    5065 cpu_manager.go:414] "containerMap: removing container" podUID="260a295b-a834-4498-9846-db028879c221" containerName="revoke-keys"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420679    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260a295b-a834-4498-9846-db028879c221" containerName="revoke-keys"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420729    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fa761428-0a3c-41a3-8ea0-6a084ad3d07f" containerName="revoke-keys"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420737    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa761428-0a3c-41a3-8ea0-6a084ad3d07f" containerName="revoke-keys"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420879    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa761428-0a3c-41a3-8ea0-6a084ad3d07f" containerName="revoke-keys"
Oct 30 18:47:37.423143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.420891    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="260a295b-a834-4498-9846-db028879c221" containerName="revoke-keys"
Oct 30 18:47:37.436124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.435704    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.474245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.474217    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"]
Oct 30 18:47:37.475226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.475204    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:47:37.509111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.501334    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:37.509111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.503947    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxts\" (UniqueName: \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.509111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.504054    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.509111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.504108    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.564278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.564247    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:37.564443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.564291    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:37.634118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.607885    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxts\" (UniqueName: \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.634555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.634517    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.634737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.634719    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.642160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.639976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.642160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.641411    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.647679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.647649    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxts\" (UniqueName: \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\") pod \"maas-api-revoke-sub-f1e1bae1-xlvdr\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:37.655726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.655700    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:37.655823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.655736    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:37.761240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:37.761215    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:38.171123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.171057    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"]
Oct 30 18:47:38.564313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.564285    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:38.564764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.564341    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:38.655686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.655644    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:38.655839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.655724    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:38.753768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:38.753735    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr" event={"ID":"5b788440-3bf0-4f25-9895-4e60e92c8633","Type":"ContainerStarted","Data":"5efda4643dd6dd4c27b0c128a00f197135642282db2a6d4ea79da623077ddab7"}
Oct 30 18:47:39.564874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.564840    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 18:47:39.565285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.564889    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 18:47:39.655568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.655533    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 30 18:47:39.655710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.655593    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 30 18:47:39.763686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.763643    5065 generic.go:356] "Generic (PLEG): container finished" podID="5b788440-3bf0-4f25-9895-4e60e92c8633" containerID="2097e77c1312c6076749b1ecac9999ec9f4714db9766c2e3e292bff59cdf6291" exitCode=0
Oct 30 18:47:39.763864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:39.763725    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr" event={"ID":"5b788440-3bf0-4f25-9895-4e60e92c8633","Type":"ContainerDied","Data":"2097e77c1312c6076749b1ecac9999ec9f4714db9766c2e3e292bff59cdf6291"}
Oct 30 18:47:40.571653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.571605    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:40.573541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.573515    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:40.574671 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.574644    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 18:47:40.662056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.662020    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:40.662849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.662827    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:40.665245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:40.665226    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 18:47:41.023433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.023404    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:41.033832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.033803    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\" (UniqueName: \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\") pod \"5b788440-3bf0-4f25-9895-4e60e92c8633\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") "
Oct 30 18:47:41.033983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.033903    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\") pod \"5b788440-3bf0-4f25-9895-4e60e92c8633\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") "
Oct 30 18:47:41.034059 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.034041    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\") pod \"5b788440-3bf0-4f25-9895-4e60e92c8633\" (UID: \"5b788440-3bf0-4f25-9895-4e60e92c8633\") "
Oct 30 18:47:41.034341 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.034304    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca" pod "5b788440-3bf0-4f25-9895-4e60e92c8633" (UID: "5b788440-3bf0-4f25-9895-4e60e92c8633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:41.034630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.034608    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:41.036177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.036149    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts" pod "5b788440-3bf0-4f25-9895-4e60e92c8633" (UID: "5b788440-3bf0-4f25-9895-4e60e92c8633"). InnerVolumeSpecName "kube-api-access-8jxts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:41.036733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.036714    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca" pod "5b788440-3bf0-4f25-9895-4e60e92c8633" (UID: "5b788440-3bf0-4f25-9895-4e60e92c8633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:41.135597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.135521    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b788440-3bf0-4f25-9895-4e60e92c8633-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:41.135597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.135551    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jxts\" (UniqueName: \"kubernetes.io/projected/5b788440-3bf0-4f25-9895-4e60e92c8633-kube-api-access-8jxts\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:41.807079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.807049    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"
Oct 30 18:47:41.811150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.807741    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr" event={"ID":"5b788440-3bf0-4f25-9895-4e60e92c8633","Type":"ContainerDied","Data":"5efda4643dd6dd4c27b0c128a00f197135642282db2a6d4ea79da623077ddab7"}
Oct 30 18:47:41.811289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:41.811185    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5efda4643dd6dd4c27b0c128a00f197135642282db2a6d4ea79da623077ddab7"
Oct 30 18:47:42.360106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.343351    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 18:47:42.371111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.361248    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5b788440-3bf0-4f25-9895-4e60e92c8633" containerName="revoke-keys"
Oct 30 18:47:42.402210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.402168    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b788440-3bf0-4f25-9895-4e60e92c8633" containerName="revoke-keys"
Oct 30 18:47:42.402650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.402634    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b788440-3bf0-4f25-9895-4e60e92c8633" containerName="revoke-keys"
Oct 30 18:47:42.453123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.437103    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.454935 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.454900    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.455426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.455404    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.455558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.455544    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.455696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.455682    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.455819 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.455804    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.455956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.455943    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.456055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.456042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4hh\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.456463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.456443    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.456614 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.456597    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.516218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.512135    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.557951    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558019    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558128    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558212    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558270    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558302    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558348    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.569451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.558381    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4hh\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.577338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.576571    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.577338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.576841    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.577338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.577113    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.577590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.577365    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.595447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.582363    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.688189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.684340    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.688189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.685019    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4hh\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.704654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.704598    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.776112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.775367    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:42.903114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:42.902647    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:44.324990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.302785    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.373112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.329185    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.373112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.330827    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.373112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.331047    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:47:44.540320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.540171    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:44.540320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.540308    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:44.552121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.544160    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:47:44.554484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.554455    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 18:47:44.633889 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.633263    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.664801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.642409    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:47:44.714152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:44.704686    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.226125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.220289    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.328760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.303526    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.328760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.304301    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:47:45.335124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.330041    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.335124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.330465    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:45.337877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.335901    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.337877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.336239    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:47:45.337877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.336542    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:47:45.337877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.336598    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:47:45.366001 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.365919    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" event={"ID":"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0","Type":"ContainerStarted","Data":"aaefb2703f9ccaec419c900e3b06d595a87bc0fa689201dc819e62abba65820a"}
Oct 30 18:47:45.585034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.584315    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:47:45.637291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.635179    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.637291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.635891    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:47:45.766107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.766049    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.766712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.766689    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:47:45.780028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.779994    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 30 18:47:45.780332 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.780304    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.780777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:45.780756    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:47:46.274110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.257960    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:47:46.306404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.306364    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:46.364114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.337809    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:46.364114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.338366    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:47:46.389946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.366507    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:47:46.389946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.367502    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:47:46.426750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.418046    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" event={"ID":"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0","Type":"ContainerStarted","Data":"6ac24efb998539c7e8310ac51e916b26a42b2dc28840ca1fe2cd67553058fb13"}
Oct 30 18:47:46.496111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.482706    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:47:46.496111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.484318    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:47:46.504294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.501560    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:47:46.515983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.514832    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podStartSLOduration=4.514814198 podStartE2EDuration="4.514814198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:46.478846225 +0000 UTC m=+886.742311190" watchObservedRunningTime="2026-10-30 18:47:46.514814198 +0000 UTC m=+886.778279148"
Oct 30 18:47:46.921563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.921528    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:46.922725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.922699    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:46.922888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:46.922865    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:47.910112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:47.906381    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:47.910112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:47.906437    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:48.908436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:48.908399    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:48.909530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:48.909500    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:49.908859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:49.908413    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:49.908859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:49.908473    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:50.322745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.322705    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"]
Oct 30 18:47:50.327227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.327206    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.339636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.339612    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:50.340447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.340428    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:47:50.343587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.342511    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"]
Oct 30 18:47:50.417962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.417923    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjb5z\" (UniqueName: \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.419885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.419859    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.420042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.419946    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.520862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.520824    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.521053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.520908    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.521053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.520999    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjb5z\" (UniqueName: \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.523702 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.523657    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.526515 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.526492    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.537987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.537962    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjb5z\" (UniqueName: \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\") pod \"maas-api-revoke-sub-772b3bcb-n82n2\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.647113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.646999    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:50.911506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.911415    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:50.912110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:50.912050    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:51.026930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.026901    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"]
Oct 30 18:47:51.300111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.299242    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"]
Oct 30 18:47:51.316376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.316345    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.316869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.316836    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"]
Oct 30 18:47:51.452432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.452403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kpc\" (UniqueName: \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.452547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.452529    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.452758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.452736    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.460007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.459979    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2" event={"ID":"5fd78a9a-d401-4a80-8b0b-cde6de268bd1","Type":"ContainerStarted","Data":"266e2c50563c60d41b3adfcdf8750d14dc37775d0e6f65c050f6712bc031e434"}
Oct 30 18:47:51.460125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.460038    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2" event={"ID":"5fd78a9a-d401-4a80-8b0b-cde6de268bd1","Type":"ContainerStarted","Data":"44da33a2cd9d23ae028fa7cf3c03b80b02210a21cfd79f6db138f7511eec1a22"}
Oct 30 18:47:51.524451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.524381    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2" podStartSLOduration=1.524360739 podStartE2EDuration="1.524360739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:51.518365161 +0000 UTC m=+891.781830110" watchObservedRunningTime="2026-10-30 18:47:51.524360739 +0000 UTC m=+891.787825688"
Oct 30 18:47:51.554405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.554376    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.554657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.554621    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.554745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.554678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kpc\" (UniqueName: \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.555154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.555127    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.557145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.557127    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.570475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.570458    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kpc\" (UniqueName: \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\") pod \"maas-api-revoke-sub-84d2ece6-bvwl6\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.640300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.640266    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:51.912042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.911864    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:51.912042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.911917    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:51.965805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:51.965776    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"]
Oct 30 18:47:51.966434 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:51.966413    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68e66d7_4067_470a_bfd0_465009e59612.slice/crio-24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88 WatchSource:0}: Error finding container 24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88: Status 404 returned error can't find the container with id 24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88
Oct 30 18:47:52.470146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.469924    5065 generic.go:356] "Generic (PLEG): container finished" podID="5fd78a9a-d401-4a80-8b0b-cde6de268bd1" containerID="266e2c50563c60d41b3adfcdf8750d14dc37775d0e6f65c050f6712bc031e434" exitCode=0
Oct 30 18:47:52.470146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.469966    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2" event={"ID":"5fd78a9a-d401-4a80-8b0b-cde6de268bd1","Type":"ContainerDied","Data":"266e2c50563c60d41b3adfcdf8750d14dc37775d0e6f65c050f6712bc031e434"}
Oct 30 18:47:52.471906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.471877    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6" event={"ID":"f68e66d7-4067-470a-bfd0-465009e59612","Type":"ContainerStarted","Data":"73b1ccdc407c87388670e14f2ed18187a7729033ab3b91820aaa339617e1d0b9"}
Oct 30 18:47:52.471996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.471931    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6" event={"ID":"f68e66d7-4067-470a-bfd0-465009e59612","Type":"ContainerStarted","Data":"24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88"}
Oct 30 18:47:52.546022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.545929    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6" podStartSLOduration=1.5459140869999999 podStartE2EDuration="1.545914087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:52.526737692 +0000 UTC m=+892.790202671" watchObservedRunningTime="2026-10-30 18:47:52.545914087 +0000 UTC m=+892.809379083"
Oct 30 18:47:52.904587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.904515    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 18:47:52.904587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:52.904565    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 18:47:53.070778 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:47:53.070720    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:47:53.486256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.486218    5065 generic.go:356] "Generic (PLEG): container finished" podID="f68e66d7-4067-470a-bfd0-465009e59612" containerID="73b1ccdc407c87388670e14f2ed18187a7729033ab3b91820aaa339617e1d0b9" exitCode=0
Oct 30 18:47:53.486905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.486875    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6" event={"ID":"f68e66d7-4067-470a-bfd0-465009e59612","Type":"ContainerDied","Data":"73b1ccdc407c87388670e14f2ed18187a7729033ab3b91820aaa339617e1d0b9"}
Oct 30 18:47:53.689872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.689849    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:53.714536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.714508    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\" (UniqueName: \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\") pod \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") "
Oct 30 18:47:53.714648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.714577    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\") pod \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") "
Oct 30 18:47:53.714648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.714645    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\") pod \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\" (UID: \"5fd78a9a-d401-4a80-8b0b-cde6de268bd1\") "
Oct 30 18:47:53.715012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.714987    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca" pod "5fd78a9a-d401-4a80-8b0b-cde6de268bd1" (UID: "5fd78a9a-d401-4a80-8b0b-cde6de268bd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:53.716843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.716822    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca" pod "5fd78a9a-d401-4a80-8b0b-cde6de268bd1" (UID: "5fd78a9a-d401-4a80-8b0b-cde6de268bd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:53.717032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.717001    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z" pod "5fd78a9a-d401-4a80-8b0b-cde6de268bd1" (UID: "5fd78a9a-d401-4a80-8b0b-cde6de268bd1"). InnerVolumeSpecName "kube-api-access-cjb5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:53.816204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.816172    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:53.816204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.816201    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjb5z\" (UniqueName: \"kubernetes.io/projected/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-kube-api-access-cjb5z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:53.816424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.816213    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd78a9a-d401-4a80-8b0b-cde6de268bd1-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:53.909693 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.909665    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:53.910039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.910020    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:53.910696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:53.910679    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 18:47:54.495834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.495802    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"
Oct 30 18:47:54.496252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.495831    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2" event={"ID":"5fd78a9a-d401-4a80-8b0b-cde6de268bd1","Type":"ContainerDied","Data":"44da33a2cd9d23ae028fa7cf3c03b80b02210a21cfd79f6db138f7511eec1a22"}
Oct 30 18:47:54.496252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.495879    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44da33a2cd9d23ae028fa7cf3c03b80b02210a21cfd79f6db138f7511eec1a22"
Oct 30 18:47:54.660136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.660074    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:54.727508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.727479    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\") pod \"f68e66d7-4067-470a-bfd0-465009e59612\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") "
Oct 30 18:47:54.728404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.728381    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\" (UniqueName: \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\") pod \"f68e66d7-4067-470a-bfd0-465009e59612\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") "
Oct 30 18:47:54.728550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.728323    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca" pod "f68e66d7-4067-470a-bfd0-465009e59612" (UID: "f68e66d7-4067-470a-bfd0-465009e59612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:54.728550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.728518    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\") pod \"f68e66d7-4067-470a-bfd0-465009e59612\" (UID: \"f68e66d7-4067-470a-bfd0-465009e59612\") "
Oct 30 18:47:54.729303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.729203    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:54.731054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.731030    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc" pod "f68e66d7-4067-470a-bfd0-465009e59612" (UID: "f68e66d7-4067-470a-bfd0-465009e59612"). InnerVolumeSpecName "kube-api-access-g4kpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:54.732262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.732203    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca" pod "f68e66d7-4067-470a-bfd0-465009e59612" (UID: "f68e66d7-4067-470a-bfd0-465009e59612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:54.830257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.830229    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4kpc\" (UniqueName: \"kubernetes.io/projected/f68e66d7-4067-470a-bfd0-465009e59612-kube-api-access-g4kpc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:54.830257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:54.830257    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68e66d7-4067-470a-bfd0-465009e59612-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:55.508189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.507361    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"
Oct 30 18:47:55.508189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.507364    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6" event={"ID":"f68e66d7-4067-470a-bfd0-465009e59612","Type":"ContainerDied","Data":"24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88"}
Oct 30 18:47:55.508189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.507513    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ee9811b64f8a1f6a0ad501cf809ea308bafd463ce56819c8c733c04358bf88"
Oct 30 18:47:55.801464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.801436    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"]
Oct 30 18:47:55.802023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802010    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f68e66d7-4067-470a-bfd0-465009e59612" containerName="revoke-keys"
Oct 30 18:47:55.802065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802024    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68e66d7-4067-470a-bfd0-465009e59612" containerName="revoke-keys"
Oct 30 18:47:55.802065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802050    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5fd78a9a-d401-4a80-8b0b-cde6de268bd1" containerName="revoke-keys"
Oct 30 18:47:55.802065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802057    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd78a9a-d401-4a80-8b0b-cde6de268bd1" containerName="revoke-keys"
Oct 30 18:47:55.802206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802199    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68e66d7-4067-470a-bfd0-465009e59612" containerName="revoke-keys"
Oct 30 18:47:55.802239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.802217    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd78a9a-d401-4a80-8b0b-cde6de268bd1" containerName="revoke-keys"
Oct 30 18:47:55.805954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.805939    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.812760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.812723    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:55.812860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.812758    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:47:55.818208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.818177    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"]
Oct 30 18:47:55.844037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.844011    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.844260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.844235    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.844375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.844356    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ckl\" (UniqueName: \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.945898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.945875    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.946037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.945973    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ckl\" (UniqueName: \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.946117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.946054    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.946774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.946701    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.948483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.948459    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:55.959774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:55.959746    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ckl\" (UniqueName: \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\") pod \"maas-api-revoke-sub-e39a61ed-87qct\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:56.123715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:56.123637    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:56.408459 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:56.408425    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0980cfb_4d85_4d54_b7a3_5f149d81956f.slice/crio-13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b WatchSource:0}: Error finding container 13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b: Status 404 returned error can't find the container with id 13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b
Oct 30 18:47:56.409433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:56.409410    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"]
Oct 30 18:47:56.537157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:56.537119    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct" event={"ID":"f0980cfb-4d85-4d54-b7a3-5f149d81956f","Type":"ContainerStarted","Data":"13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b"}
Oct 30 18:47:57.487147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.487112    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"]
Oct 30 18:47:57.492192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.492168    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.501378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.501357    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"]
Oct 30 18:47:57.546979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.546901    5065 generic.go:356] "Generic (PLEG): container finished" podID="f0980cfb-4d85-4d54-b7a3-5f149d81956f" containerID="f065cf72381805abceee3e4cfe3a1014875c9cf6e626d552e38150ac26426a02" exitCode=0
Oct 30 18:47:57.546979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.546954    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct" event={"ID":"f0980cfb-4d85-4d54-b7a3-5f149d81956f","Type":"ContainerDied","Data":"f065cf72381805abceee3e4cfe3a1014875c9cf6e626d552e38150ac26426a02"}
Oct 30 18:47:57.569159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.569126    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgpw\" (UniqueName: \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.569307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.569271    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.569384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.569364    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.670638 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.670604    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.670825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.670651    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.670825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.670751    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgpw\" (UniqueName: \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.671334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.671312    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.673453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.673432    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.684969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.684946    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgpw\" (UniqueName: \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\") pod \"maas-api-revoke-sub-76db0ccd-c9v62\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:57.824491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:57.824422    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:47:58.113279 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:47:58.113247    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870001e6_feb9_4909_961b_953d412bc413.slice/crio-7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97 WatchSource:0}: Error finding container 7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97: Status 404 returned error can't find the container with id 7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97
Oct 30 18:47:58.114864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.114836    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"]
Oct 30 18:47:58.555444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.555403    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62" event={"ID":"870001e6-feb9-4909-961b-953d412bc413","Type":"ContainerStarted","Data":"7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97"}
Oct 30 18:47:58.730689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.730661    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:47:58.786577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.786529    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\" (UniqueName: \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\") pod \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") "
Oct 30 18:47:58.786747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.786591    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\") pod \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") "
Oct 30 18:47:58.786747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.786719    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\") pod \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\" (UID: \"f0980cfb-4d85-4d54-b7a3-5f149d81956f\") "
Oct 30 18:47:58.788662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.787120    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca" pod "f0980cfb-4d85-4d54-b7a3-5f149d81956f" (UID: "f0980cfb-4d85-4d54-b7a3-5f149d81956f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:58.788662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.787571    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:58.796506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.789379    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca" pod "f0980cfb-4d85-4d54-b7a3-5f149d81956f" (UID: "f0980cfb-4d85-4d54-b7a3-5f149d81956f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:58.796506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.789415    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl" pod "f0980cfb-4d85-4d54-b7a3-5f149d81956f" (UID: "f0980cfb-4d85-4d54-b7a3-5f149d81956f"). InnerVolumeSpecName "kube-api-access-54ckl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:58.889195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.889143    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ckl\" (UniqueName: \"kubernetes.io/projected/f0980cfb-4d85-4d54-b7a3-5f149d81956f-kube-api-access-54ckl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:58.889195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:58.889171    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0980cfb-4d85-4d54-b7a3-5f149d81956f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:59.568484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:59.568391    5065 generic.go:356] "Generic (PLEG): container finished" podID="870001e6-feb9-4909-961b-953d412bc413" containerID="964cfd3be904bbf2997d426123ceb79222d46fc413165e3c16e3f6050bdded50" exitCode=0
Oct 30 18:47:59.568936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:59.568522    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62" event={"ID":"870001e6-feb9-4909-961b-953d412bc413","Type":"ContainerDied","Data":"964cfd3be904bbf2997d426123ceb79222d46fc413165e3c16e3f6050bdded50"}
Oct 30 18:47:59.587289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:59.586855    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct" event={"ID":"f0980cfb-4d85-4d54-b7a3-5f149d81956f","Type":"ContainerDied","Data":"13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b"}
Oct 30 18:47:59.587289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:59.586907    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c8d1377570d2aa59c34834294fe1f1ea1c19814c5619491cd5d6892d2c3b8b"
Oct 30 18:47:59.587289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:47:59.586975    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"
Oct 30 18:48:01.343322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:01.343277    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:01.664306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:01.647778    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:01.664306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:01.648168    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:48:01.664306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:01.648475    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:48:01.664306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:01.662554    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.350305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.347167    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:48:02.350305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.349170    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.380264    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.396498    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.396550    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.396942    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.396999    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"]
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.398198    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f0980cfb-4d85-4d54-b7a3-5f149d81956f" containerName="revoke-keys"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.398216    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0980cfb-4d85-4d54-b7a3-5f149d81956f" containerName="revoke-keys"
Oct 30 18:48:02.401794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.398390    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0980cfb-4d85-4d54-b7a3-5f149d81956f" containerName="revoke-keys"
Oct 30 18:48:02.450308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.447436    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"]
Oct 30 18:48:02.450308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.447553    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.503111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.503032    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.624169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.621683    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.624169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.621750    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pnz\" (UniqueName: \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.624169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.621806    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.679116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.677389    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.679116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.677691    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.679116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.677877    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.728143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.724155    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:02.728143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.724248    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:02.728143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.724314    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:02.770113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.765872    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.770113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.765942    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pnz\" (UniqueName: \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.770113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.765985    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.770113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.766930    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.900489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.893962    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:02.900489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.894838    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:48:02.988858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.971188    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pnz\" (UniqueName: \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:02.989115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.988507    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\") pod \"870001e6-feb9-4909-961b-953d412bc413\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") "
Oct 30 18:48:02.989402 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.989384    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\") pod \"870001e6-feb9-4909-961b-953d412bc413\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") "
Oct 30 18:48:03.130109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.097526    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\" (UniqueName: \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\") pod \"870001e6-feb9-4909-961b-953d412bc413\" (UID: \"870001e6-feb9-4909-961b-953d412bc413\") "
Oct 30 18:48:03.130109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:02.990035    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca" pod "870001e6-feb9-4909-961b-953d412bc413" (UID: "870001e6-feb9-4909-961b-953d412bc413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:03.130109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.091141    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:48:03.197163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.081828    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:48:03.197163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.194751    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:48:03.216676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.199583    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870001e6-feb9-4909-961b-953d412bc413-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:03.216676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.214626    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca" pod "870001e6-feb9-4909-961b-953d412bc413" (UID: "870001e6-feb9-4909-961b-953d412bc413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:03.239125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.231460    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4672b909-qvms8\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:03.295113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.266252    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw" pod "870001e6-feb9-4909-961b-953d412bc413" (UID: "870001e6-feb9-4909-961b-953d412bc413"). InnerVolumeSpecName "kube-api-access-jvgpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:03.295723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.295695    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:03.318112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.297340    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:03.318112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.309715    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870001e6-feb9-4909-961b-953d412bc413-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:03.318112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.309739    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgpw\" (UniqueName: \"kubernetes.io/projected/870001e6-feb9-4909-961b-953d412bc413-kube-api-access-jvgpw\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:03.318112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.315634    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:48:03.360111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.319372    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:03.381686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.381064    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:03.381686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.381328    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:03.419452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.419427    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:03.510206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.508692    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:03.539972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.539943    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"
Oct 30 18:48:03.542496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.542467    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62" event={"ID":"870001e6-feb9-4909-961b-953d412bc413","Type":"ContainerDied","Data":"7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97"}
Oct 30 18:48:03.542692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.542677    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7822c4ca19efffbf70f340735a35959401963584c40c4bf9fb56efcf8d0f97"
Oct 30 18:48:03.930441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:03.930392    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"]
Oct 30 18:48:04.551812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:04.551732    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8" event={"ID":"cb885de6-dcde-4d90-a0e9-364e37f80aa7","Type":"ContainerStarted","Data":"25b7f4bf27f7d268445a0809d02a16de8224161ec9b70059e5bf7f1e51053e22"}
Oct 30 18:48:04.551812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:04.551795    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8" event={"ID":"cb885de6-dcde-4d90-a0e9-364e37f80aa7","Type":"ContainerStarted","Data":"0458abb56be069d44f751421423ce0a5450aaa25f88cf4f24c18c28973263aa7"}
Oct 30 18:48:04.590186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:04.590058    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8" podStartSLOduration=3.590041099 podStartE2EDuration="3.590041099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:04.587861127 +0000 UTC m=+904.851326076" watchObservedRunningTime="2026-10-30 18:48:04.590041099 +0000 UTC m=+904.853506048"
Oct 30 18:48:05.562847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:05.562811    5065 generic.go:356] "Generic (PLEG): container finished" podID="cb885de6-dcde-4d90-a0e9-364e37f80aa7" containerID="25b7f4bf27f7d268445a0809d02a16de8224161ec9b70059e5bf7f1e51053e22" exitCode=0
Oct 30 18:48:05.563265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:05.562900    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8" event={"ID":"cb885de6-dcde-4d90-a0e9-364e37f80aa7","Type":"ContainerDied","Data":"25b7f4bf27f7d268445a0809d02a16de8224161ec9b70059e5bf7f1e51053e22"}
Oct 30 18:48:06.104270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.104237    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"]
Oct 30 18:48:06.104860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.104844    5065 cpu_manager.go:414] "containerMap: removing container" podUID="870001e6-feb9-4909-961b-953d412bc413" containerName="revoke-keys"
Oct 30 18:48:06.104860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.104860    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870001e6-feb9-4909-961b-953d412bc413" containerName="revoke-keys"
Oct 30 18:48:06.105008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.104944    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="870001e6-feb9-4909-961b-953d412bc413" containerName="revoke-keys"
Oct 30 18:48:06.108919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.108897    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.165308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.165282    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"]
Oct 30 18:48:06.187570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.187541    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7j96\" (UniqueName: \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.187704 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.187592    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.187890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.187873    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.289146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.289113    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7j96\" (UniqueName: \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.289310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.289166    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.289410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.289391    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.290070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.290045    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.291799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.291778    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.303563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.303543    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7j96\" (UniqueName: \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\") pod \"maas-api-revoke-sub-8cf63931-cp4fr\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.424697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.424630    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:06.741574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.741544    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"]
Oct 30 18:48:06.749883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.749863    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:06.797759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.797736    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\") pod \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") "
Oct 30 18:48:06.797863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.797768    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\") pod \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") "
Oct 30 18:48:06.797934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.797904    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\" (UniqueName: \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\") pod \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\" (UID: \"cb885de6-dcde-4d90-a0e9-364e37f80aa7\") "
Oct 30 18:48:06.798168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.798144    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca" pod "cb885de6-dcde-4d90-a0e9-364e37f80aa7" (UID: "cb885de6-dcde-4d90-a0e9-364e37f80aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:06.798490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.798457    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:06.880174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.880145    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca" pod "cb885de6-dcde-4d90-a0e9-364e37f80aa7" (UID: "cb885de6-dcde-4d90-a0e9-364e37f80aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:06.880174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.880168    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz" pod "cb885de6-dcde-4d90-a0e9-364e37f80aa7" (UID: "cb885de6-dcde-4d90-a0e9-364e37f80aa7"). InnerVolumeSpecName "kube-api-access-x4pnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:06.899690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.899621    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pnz\" (UniqueName: \"kubernetes.io/projected/cb885de6-dcde-4d90-a0e9-364e37f80aa7-kube-api-access-x4pnz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:06.899690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:06.899656    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb885de6-dcde-4d90-a0e9-364e37f80aa7-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:07.109654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.109616    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"]
Oct 30 18:48:07.110523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.110493    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cb885de6-dcde-4d90-a0e9-364e37f80aa7" containerName="revoke-keys"
Oct 30 18:48:07.110523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.110515    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb885de6-dcde-4d90-a0e9-364e37f80aa7" containerName="revoke-keys"
Oct 30 18:48:07.110813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.110637    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb885de6-dcde-4d90-a0e9-364e37f80aa7" containerName="revoke-keys"
Oct 30 18:48:07.115623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.115593    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.139801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.139762    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"]
Oct 30 18:48:07.203988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.203966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf72\" (UniqueName: \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.204173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.204155    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.204227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.204204    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.305314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.305276    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.305427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.305409    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf72\" (UniqueName: \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.305593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.305577    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.306039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.306014    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.308509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.308486    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.357919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.357889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf72\" (UniqueName: \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\") pod \"maas-api-revoke-sub-0a5b5294-5cqmh\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.432380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.432347    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:07.599169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.598893    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"
Oct 30 18:48:07.599169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.598921    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8" event={"ID":"cb885de6-dcde-4d90-a0e9-364e37f80aa7","Type":"ContainerDied","Data":"0458abb56be069d44f751421423ce0a5450aaa25f88cf4f24c18c28973263aa7"}
Oct 30 18:48:07.600568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.599658    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0458abb56be069d44f751421423ce0a5450aaa25f88cf4f24c18c28973263aa7"
Oct 30 18:48:07.601014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.600983    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr" event={"ID":"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640","Type":"ContainerStarted","Data":"c3388f291222306f9b5096b35352d58c2850c3703d5f6c21b33da776aa93dfb0"}
Oct 30 18:48:07.601365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.601343    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr" event={"ID":"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640","Type":"ContainerStarted","Data":"515407501d634af5af7136caad1bd6f9645f8a1939a77dd3711e3dfd06650979"}
Oct 30 18:48:07.635415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.635372    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr" podStartSLOduration=1.635359512 podStartE2EDuration="1.635359512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:07.634816245 +0000 UTC m=+907.898281210" watchObservedRunningTime="2026-10-30 18:48:07.635359512 +0000 UTC m=+907.898824472"
Oct 30 18:48:07.752133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.752110    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"]
Oct 30 18:48:07.783932 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:07.783901    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e6bdcc_2be8_4a64_aa59_b780e060a1b5.slice/crio-44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377 WatchSource:0}: Error finding container 44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377: Status 404 returned error can't find the container with id 44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377
Oct 30 18:48:07.790618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.790593    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"]
Oct 30 18:48:07.795996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.795979    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:07.810425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.810397    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"]
Oct 30 18:48:07.914157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.914026    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:07.914157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.914074    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcgj\" (UniqueName: \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:07.914454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:07.914424    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.016187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.016156    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.016309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.016199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcgj\" (UniqueName: \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.016309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.016265    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.016917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.016888    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.018849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.018825    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.030739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.030706    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcgj\" (UniqueName: \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\") pod \"maas-api-revoke-sub-02b3ea6f-bjch7\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.113353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.113333    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:08.404753 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.404720    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"]
Oct 30 18:48:08.405312 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:08.405282    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320b207b_ad97_46a2_9828_6ec59dc3d08b.slice/crio-8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c WatchSource:0}: Error finding container 8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c: Status 404 returned error can't find the container with id 8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c
Oct 30 18:48:08.614533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.614494    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh" event={"ID":"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5","Type":"ContainerStarted","Data":"a264e1ce4de38840c63a3d65bc89d8ceb42ec6ff00ff8a177c6578958ba59b56"}
Oct 30 18:48:08.614721 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.614563    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh" event={"ID":"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5","Type":"ContainerStarted","Data":"44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377"}
Oct 30 18:48:08.616362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.616332    5065 generic.go:356] "Generic (PLEG): container finished" podID="f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" containerID="c3388f291222306f9b5096b35352d58c2850c3703d5f6c21b33da776aa93dfb0" exitCode=0
Oct 30 18:48:08.616509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.616410    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr" event={"ID":"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640","Type":"ContainerDied","Data":"c3388f291222306f9b5096b35352d58c2850c3703d5f6c21b33da776aa93dfb0"}
Oct 30 18:48:08.618061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.618026    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7" event={"ID":"320b207b-ad97-46a2-9828-6ec59dc3d08b","Type":"ContainerStarted","Data":"8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c"}
Oct 30 18:48:08.653747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:08.653696    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh" podStartSLOduration=1.653682561 podStartE2EDuration="1.653682561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:08.650647028 +0000 UTC m=+908.914111995" watchObservedRunningTime="2026-10-30 18:48:08.653682561 +0000 UTC m=+908.917147509"
Oct 30 18:48:09.632244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.632166    5065 generic.go:356] "Generic (PLEG): container finished" podID="e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" containerID="a264e1ce4de38840c63a3d65bc89d8ceb42ec6ff00ff8a177c6578958ba59b56" exitCode=0
Oct 30 18:48:09.632641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.632251    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh" event={"ID":"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5","Type":"ContainerDied","Data":"a264e1ce4de38840c63a3d65bc89d8ceb42ec6ff00ff8a177c6578958ba59b56"}
Oct 30 18:48:09.634496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.634464    5065 generic.go:356] "Generic (PLEG): container finished" podID="320b207b-ad97-46a2-9828-6ec59dc3d08b" containerID="5a9db873f00fcca123e110f3aad4d20458b1e1bd6ecc2d83124278a0b27a082e" exitCode=0
Oct 30 18:48:09.635065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.635039    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7" event={"ID":"320b207b-ad97-46a2-9828-6ec59dc3d08b","Type":"ContainerDied","Data":"5a9db873f00fcca123e110f3aad4d20458b1e1bd6ecc2d83124278a0b27a082e"}
Oct 30 18:48:09.818972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.818948    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:09.840181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.840154    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\") pod \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") "
Oct 30 18:48:09.840352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.840257    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\" (UniqueName: \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\") pod \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") "
Oct 30 18:48:09.840352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.840297    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\") pod \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\" (UID: \"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640\") "
Oct 30 18:48:09.840759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.840732    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca" pod "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" (UID: "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:09.841026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.840925    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:09.844248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.844221    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca" pod "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" (UID: "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:09.844359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.844246    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96" pod "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" (UID: "f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640"). InnerVolumeSpecName "kube-api-access-l7j96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:09.942461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.942437    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:09.942461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.942460    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7j96\" (UniqueName: \"kubernetes.io/projected/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640-kube-api-access-l7j96\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:09.968931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.968908    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:48:09.969590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.969575    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" containerName="revoke-keys"
Oct 30 18:48:09.969648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.969592    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" containerName="revoke-keys"
Oct 30 18:48:09.969692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.969683    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" containerName="revoke-keys"
Oct 30 18:48:09.973320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:09.973304    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.001137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.000774    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:48:10.008182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.006105    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:48:10.010805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.010788    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.038512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.038481    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:48:10.047515 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047487    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047519    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047553    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047571    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047592    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047613    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047630    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb48t\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047653    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047780    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047797    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047812    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047845    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9bv\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047868    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047885    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047899    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.047930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047928    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.048405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047943    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.048405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.047968    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149076    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149137    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149155    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149184    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9bv\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149207    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149226    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149251    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149360    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149386    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149421    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149468    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149493    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149549    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149591    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149627    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149663    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149679    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149688    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb48t\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149736    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149782    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149819    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.149944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.149836    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.150620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.150195    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.150620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.150371    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.150620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.150442    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.150953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.150648    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.150953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.150680    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.152491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.152466    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.152597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.152561    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.152650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.152598    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.152777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.152757    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.163337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.163284    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb48t\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.166860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.166836    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9bv\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.167031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.167014    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.167121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.167080    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.287832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.287796    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:10.339392 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.339365    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:10.635949 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:10.635918    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd383ad5a_1fa9_43f3_87aa_1b17ef345961.slice/crio-d5ac65a208c6f33a0820f25054174956efaefca40fb0f43b2a4ba1cb59e5f88f WatchSource:0}: Error finding container d5ac65a208c6f33a0820f25054174956efaefca40fb0f43b2a4ba1cb59e5f88f: Status 404 returned error can't find the container with id d5ac65a208c6f33a0820f25054174956efaefca40fb0f43b2a4ba1cb59e5f88f
Oct 30 18:48:10.638251 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.638221    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:48:10.639053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.639020    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.639139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.639117    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.639192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.639163    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.646358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.646335    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"
Oct 30 18:48:10.646568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.646471    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr" event={"ID":"f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640","Type":"ContainerDied","Data":"515407501d634af5af7136caad1bd6f9645f8a1939a77dd3711e3dfd06650979"}
Oct 30 18:48:10.646681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.646603    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="515407501d634af5af7136caad1bd6f9645f8a1939a77dd3711e3dfd06650979"
Oct 30 18:48:10.718725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.718698    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:48:10.736498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.735763    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.736498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.735850    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.736498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.735895    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:10.853121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.853066    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:10.929393 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.929361    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:10.971183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.971141    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\") pod \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") "
Oct 30 18:48:10.972400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.972161    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\") pod \"320b207b-ad97-46a2-9828-6ec59dc3d08b\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") "
Oct 30 18:48:10.972400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.972363    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\" (UniqueName: \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\") pod \"320b207b-ad97-46a2-9828-6ec59dc3d08b\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") "
Oct 30 18:48:10.972600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.972446    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\" (UniqueName: \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\") pod \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") "
Oct 30 18:48:10.972600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.972522    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\") pod \"320b207b-ad97-46a2-9828-6ec59dc3d08b\" (UID: \"320b207b-ad97-46a2-9828-6ec59dc3d08b\") "
Oct 30 18:48:10.972600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.972567    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\") pod \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\" (UID: \"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5\") "
Oct 30 18:48:10.974100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.973381    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca" pod "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" (UID: "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:10.974100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.974042    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca" pod "320b207b-ad97-46a2-9828-6ec59dc3d08b" (UID: "320b207b-ad97-46a2-9828-6ec59dc3d08b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:10.975780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.975749    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj" pod "320b207b-ad97-46a2-9828-6ec59dc3d08b" (UID: "320b207b-ad97-46a2-9828-6ec59dc3d08b"). InnerVolumeSpecName "kube-api-access-8qcgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:10.977923 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.977872    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca" pod "320b207b-ad97-46a2-9828-6ec59dc3d08b" (UID: "320b207b-ad97-46a2-9828-6ec59dc3d08b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:10.978039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.977967    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca" pod "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" (UID: "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:10.978757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:10.978733    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72" pod "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" (UID: "e7e6bdcc-2be8-4a64-aa59-b780e060a1b5"). InnerVolumeSpecName "kube-api-access-9jf72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074670    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jf72\" (UniqueName: \"kubernetes.io/projected/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-kube-api-access-9jf72\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074702    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074712    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074721    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074731    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320b207b-ad97-46a2-9828-6ec59dc3d08b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.074748 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.074740    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcgj\" (UniqueName: \"kubernetes.io/projected/320b207b-ad97-46a2-9828-6ec59dc3d08b-kube-api-access-8qcgj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.668592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.668493    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh" event={"ID":"e7e6bdcc-2be8-4a64-aa59-b780e060a1b5","Type":"ContainerDied","Data":"44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377"}
Oct 30 18:48:11.669179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.669157    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44120f76eff5d5196d1ae8c090116b47870163c1dec83ae3ebfaed12b5a0e377"
Oct 30 18:48:11.669395 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.669377    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"
Oct 30 18:48:11.706136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.679343    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" event={"ID":"d383ad5a-1fa9-43f3-87aa-1b17ef345961","Type":"ContainerStarted","Data":"4303b046109983061a4ad82eb713dc3f3989753a2322d0eaa0b41d9c698d99b1"}
Oct 30 18:48:11.706136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.679400    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" event={"ID":"d383ad5a-1fa9-43f3-87aa-1b17ef345961","Type":"ContainerStarted","Data":"d5ac65a208c6f33a0820f25054174956efaefca40fb0f43b2a4ba1cb59e5f88f"}
Oct 30 18:48:11.706136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.704802    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" event={"ID":"f0ac827e-8d2a-482b-8596-27baeb496787","Type":"ContainerStarted","Data":"a130c48dfc6ce92efd37887f41d72052c581f8482ff6ef61da1154d9e9465bb6"}
Oct 30 18:48:11.706136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.704864    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" event={"ID":"f0ac827e-8d2a-482b-8596-27baeb496787","Type":"ContainerStarted","Data":"b3a667b72c75b8b4ab821add9d46da30720c8e51b06468f68ec502ba40243a50"}
Oct 30 18:48:11.726496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.714207    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7" event={"ID":"320b207b-ad97-46a2-9828-6ec59dc3d08b","Type":"ContainerDied","Data":"8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c"}
Oct 30 18:48:11.726496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.714264    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e027c3eaf14489397656d844bd6ed39640c4f5daacda6d170f917db2cdbf06c"
Oct 30 18:48:11.726496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.714329    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"
Oct 30 18:48:11.774648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.774602    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podStartSLOduration=2.7745892960000003 podStartE2EDuration="2.774589296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.76857304 +0000 UTC m=+912.032038004" watchObservedRunningTime="2026-10-30 18:48:11.774589296 +0000 UTC m=+912.038054244"
Oct 30 18:48:11.977682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:11.956578    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podStartSLOduration=2.956557385 podStartE2EDuration="2.956557385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.840858904 +0000 UTC m=+912.104323871" watchObservedRunningTime="2026-10-30 18:48:11.956557385 +0000 UTC m=+912.220022334"
Oct 30 18:48:12.306160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.293380    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:12.306160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.293456    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:12.306160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.293754    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:12.348185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.348148    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:12.349462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.349432    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:12.349676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:12.349650    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:13.290582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:13.290550    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:13.291230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:13.291182    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:13.345892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:13.344490    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:13.345892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:13.344560    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:14.294111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:14.293553    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:14.294111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:14.293608    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:14.347540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:14.347503    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:14.347793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:14.347765    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:15.309247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:15.305177    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:15.309247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:15.305238    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:15.367118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:15.365043    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:15.367118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:15.365133    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:15.717170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:15.716195    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.295110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.294132    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:16.295110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.294202    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:16.350113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.349133    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:16.350113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.349206    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:16.404111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.403173    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.458114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.434348    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:16.503109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.502170    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.551165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.550240    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.928166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:16.927173    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.266162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.265279    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.313111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.301116    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:17.313111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.301187    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:17.317110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.316759    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:48:17.362764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.351180    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:17.362764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.351239    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.449961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.449881    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:17.450273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.449935    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:17.490111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.462170    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.490111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.462536    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:48:17.490111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.462865    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:17.490111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.462941    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:17.534490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.532746    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.575177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.535498    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:48:17.575177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.556192    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:48:17.575177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.556546    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:17.785412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.785301    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.947111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:17.938550    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.141623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.141537    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:18.179256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.179228    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"]
Oct 30 18:48:18.203355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.203325    5065 cpu_manager.go:414] "containerMap: removing container" podUID="320b207b-ad97-46a2-9828-6ec59dc3d08b" containerName="revoke-keys"
Oct 30 18:48:18.203565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.203549    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320b207b-ad97-46a2-9828-6ec59dc3d08b" containerName="revoke-keys"
Oct 30 18:48:18.203736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.203722    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" containerName="revoke-keys"
Oct 30 18:48:18.203840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.203827    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" containerName="revoke-keys"
Oct 30 18:48:18.204117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.204079    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" containerName="revoke-keys"
Oct 30 18:48:18.204237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.204225    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="320b207b-ad97-46a2-9828-6ec59dc3d08b" containerName="revoke-keys"
Oct 30 18:48:18.236405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.236382    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"]
Oct 30 18:48:18.237010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.236668    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.269191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.269152    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.269861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.269834    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:48:18.294379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.294353    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:48:18.294723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.294703    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:48:18.295176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.295152    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:18.295343 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.295320    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:18.339113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.331502    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.339113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.331939    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:18.363291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.363257    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:18.395419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.395333    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:18.395640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.393770    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.395802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.395785    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.396016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.395999    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4qn\" (UniqueName: \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.548112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.470539    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:18.548752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.472154    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.477484    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.504280    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.569416    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.569529    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4qn\" (UniqueName: \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.504974    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.571285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.558234    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.577961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.577909    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.610890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.610861    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:18.720426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.679154    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:18.720426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.702562    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4qn\" (UniqueName: \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\") pod \"maas-api-revoke-sub-4a0ee2de-f5tc6\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.778306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.778280    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:18.826060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.825140    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:18.880115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.879596    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:18.926047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.925843    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:18.972388 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:18.972317    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:48:19.289625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.289590    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.289805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.289649    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 18:48:19.302740 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:19.301979    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffa2d77_c1bd_4d34_bba4_23d9d5c04bd5.slice/crio-07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee WatchSource:0}: Error finding container 07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee: Status 404 returned error can't find the container with id 07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee
Oct 30 18:48:19.305123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.305068    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"]
Oct 30 18:48:19.340556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.340527    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.340680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.340590    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 18:48:19.419034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:19.418988    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6" event={"ID":"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5","Type":"ContainerStarted","Data":"07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee"}
Oct 30 18:48:20.431600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:20.431566    5065 generic.go:356] "Generic (PLEG): container finished" podID="9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" containerID="8d31e8d7b5df40aa2cb9e42a4b4715cd7b89ff7d7f65983d720983cb91bc7da2" exitCode=0
Oct 30 18:48:20.432066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:20.431653    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6" event={"ID":"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5","Type":"ContainerDied","Data":"8d31e8d7b5df40aa2cb9e42a4b4715cd7b89ff7d7f65983d720983cb91bc7da2"}
Oct 30 18:48:20.510112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:20.504466    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:20.510112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:20.505963    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:20.510112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:20.507467    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:48:21.336109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:21.335240    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:21.336109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:21.335303    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:22.118244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.118101    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:48:22.123756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.123727    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:22.125794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.125773    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:48:22.125963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.125897    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.155774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.155743    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4d94b-openshift-default-dockercfg-5lxhz\""
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.244190    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\") pod \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") "
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.244310    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\" (UniqueName: \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\") pod \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") "
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.244388    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\") pod \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\" (UID: \"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5\") "
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245435    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245493    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245575    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245704    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245738    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245898    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.245966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfchh\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.246064    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.253167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.247989    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca" pod "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" (UID: "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:22.270189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.270154    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:22.273114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.272732    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:22.281029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.281008    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:48:22.288827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.288788    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn" pod "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" (UID: "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5"). InnerVolumeSpecName "kube-api-access-fz4qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:22.302144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.299211    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca" pod "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" (UID: "9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.348975    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349006    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349023    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfchh\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349079    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349158    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349196    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349237    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349285    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349313    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349382    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4qn\" (UniqueName: \"kubernetes.io/projected/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-kube-api-access-fz4qn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349393    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349405    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.349692    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.350525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.350036    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.355245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.353956    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.360243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.356585    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.360243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.359525    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.364740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.364674    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.367346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.367316    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.379531 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.379509    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfchh\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.379905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.379888    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\") pod \"e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") " pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.466652 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.466620    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:22.467491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.467469    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"
Oct 30 18:48:22.467620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.467533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6" event={"ID":"9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5","Type":"ContainerDied","Data":"07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee"}
Oct 30 18:48:22.467620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.467581    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c4121ade7937700dff5ba6be2e965df7f6c2922e1c638699012e3cb63cd2ee"
Oct 30 18:48:22.615757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.615713    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"]
Oct 30 18:48:22.616527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.616505    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" containerName="revoke-keys"
Oct 30 18:48:22.616527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.616526    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" containerName="revoke-keys"
Oct 30 18:48:22.616731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.616700    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" containerName="revoke-keys"
Oct 30 18:48:22.621554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.621536    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.644791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.644766    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"]
Oct 30 18:48:22.655395 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.655370    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.655556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.655538    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.655626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.655600    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.669813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.669792    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:48:22.669976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.669951    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:48:22.757289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.757262    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.757632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.757609    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.757734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.757685    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.758376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.758350    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.760206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.760174    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.775538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.775508    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\") pod \"maas-api-revoke-sub-bb394f71-zjxhj\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:22.797461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.797410    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:48:22.801595 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:22.801574    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc8d082_0ce1_48da_bf12_2d3b1470e8af.slice/crio-0572bc0d42367968065b5e1e6652e5ad2aa83ddacd14520e0afd06dd64c5a7b1 WatchSource:0}: Error finding container 0572bc0d42367968065b5e1e6652e5ad2aa83ddacd14520e0afd06dd64c5a7b1: Status 404 returned error can't find the container with id 0572bc0d42367968065b5e1e6652e5ad2aa83ddacd14520e0afd06dd64c5a7b1
Oct 30 18:48:22.804581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.804548    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:22.804643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.804612    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:22.804643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.804640    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:22.937195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:22.937158    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:23.071797 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:48:23.071711    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:48:23.284148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:23.281525    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"]
Oct 30 18:48:23.284148 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:23.283590    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f15acf8_d3d8_47f9_b39f_73b172ca1d81.slice/crio-f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b WatchSource:0}: Error finding container f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b: Status 404 returned error can't find the container with id f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b
Oct 30 18:48:23.489194 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:23.489158    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerStarted","Data":"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"}
Oct 30 18:48:23.489367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:23.489225    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerStarted","Data":"0572bc0d42367968065b5e1e6652e5ad2aa83ddacd14520e0afd06dd64c5a7b1"}
Oct 30 18:48:23.490813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:23.490784    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj" event={"ID":"0f15acf8-d3d8-47f9-b39f-73b172ca1d81","Type":"ContainerStarted","Data":"f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b"}
Oct 30 18:48:24.467459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.467387    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:24.473684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.473659    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:24.504540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.504475    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podStartSLOduration=3.5044565260000002 podStartE2EDuration="3.504456526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:23.531272778 +0000 UTC m=+923.794737727" watchObservedRunningTime="2026-10-30 18:48:24.504456526 +0000 UTC m=+924.767921483"
Oct 30 18:48:24.508843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.508793    5065 generic.go:356] "Generic (PLEG): container finished" podID="0f15acf8-d3d8-47f9-b39f-73b172ca1d81" containerID="27670cdbafd10eea32ad8e1ea6a5033987409029dcbd9fdc4da4790b6efbbc4c" exitCode=0
Oct 30 18:48:24.509015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.508844    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj" event={"ID":"0f15acf8-d3d8-47f9-b39f-73b172ca1d81","Type":"ContainerDied","Data":"27670cdbafd10eea32ad8e1ea6a5033987409029dcbd9fdc4da4790b6efbbc4c"}
Oct 30 18:48:24.509450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.509428    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:24.512835 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:24.512807    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:48:25.056583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.056541    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:48:25.060945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.060922    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.074798 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.074769    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t474\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.074910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.074827    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.074910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.074853    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.074964    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.075007    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.075104    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075287 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.075214    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075287 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.075275    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.075415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.075334    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.079130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.078594    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:48:25.176545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176507    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t474\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176556    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176580    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176680    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176725    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176760    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176795    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176842    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.176984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.176885    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.177807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.177778    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.178120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.178060    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.178433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.178405    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.178579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.178409    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.178579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.178454    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.180418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.180395    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.190216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.190189    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t474\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.190313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.190228    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.203740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.203718    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.403945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.403858    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:25.708269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.708240    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:48:25.712703 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:25.712676    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bbebaa_6379_4191_a6b9_51f94eb9462d.slice/crio-c93121aaecd72d115947742bbeabdb577b49cd1f7b74117a2290ae14060583a3 WatchSource:0}: Error finding container c93121aaecd72d115947742bbeabdb577b49cd1f7b74117a2290ae14060583a3: Status 404 returned error can't find the container with id c93121aaecd72d115947742bbeabdb577b49cd1f7b74117a2290ae14060583a3
Oct 30 18:48:25.714724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.714701    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:25.715556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.715523    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:25.715661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.715645    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:25.715712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.715700    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:25.784825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.784797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\") pod \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") "
Oct 30 18:48:25.785033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.785004    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\") pod \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") "
Oct 30 18:48:25.785245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.785133    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\") pod \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\" (UID: \"0f15acf8-d3d8-47f9-b39f-73b172ca1d81\") "
Oct 30 18:48:25.786126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.786066    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca" pod "0f15acf8-d3d8-47f9-b39f-73b172ca1d81" (UID: "0f15acf8-d3d8-47f9-b39f-73b172ca1d81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:25.788931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.788901    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6" pod "0f15acf8-d3d8-47f9-b39f-73b172ca1d81" (UID: "0f15acf8-d3d8-47f9-b39f-73b172ca1d81"). InnerVolumeSpecName "kube-api-access-cbrr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:25.795728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.794781    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca" pod "0f15acf8-d3d8-47f9-b39f-73b172ca1d81" (UID: "0f15acf8-d3d8-47f9-b39f-73b172ca1d81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:25.890226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.889119    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:25.890226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.889166    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:25.890226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:25.889180    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrr6\" (UniqueName: \"kubernetes.io/projected/0f15acf8-d3d8-47f9-b39f-73b172ca1d81-kube-api-access-cbrr6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:26.541676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.541648    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"
Oct 30 18:48:26.541879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.541683    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj" event={"ID":"0f15acf8-d3d8-47f9-b39f-73b172ca1d81","Type":"ContainerDied","Data":"f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b"}
Oct 30 18:48:26.541879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.541733    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e827525bea8af9c4ae15dc482ff8ea86a1e70c787133e555660116b3e9210b"
Oct 30 18:48:26.544838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.544517    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" event={"ID":"b0bbebaa-6379-4191-a6b9-51f94eb9462d","Type":"ContainerStarted","Data":"b5ab877112db8a14f71c669e38bc4e0650d238d605674ff5c1d50a00bad09f24"}
Oct 30 18:48:26.544838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.544570    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" event={"ID":"b0bbebaa-6379-4191-a6b9-51f94eb9462d","Type":"ContainerStarted","Data":"c93121aaecd72d115947742bbeabdb577b49cd1f7b74117a2290ae14060583a3"}
Oct 30 18:48:26.598113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:26.594393    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podStartSLOduration=1.594375022 podStartE2EDuration="1.594375022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:26.584639263 +0000 UTC m=+926.848104212" watchObservedRunningTime="2026-10-30 18:48:26.594375022 +0000 UTC m=+926.857839972"
Oct 30 18:48:27.411207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:27.411174    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:27.411880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:27.411857    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:27.412053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:27.412030    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:28.406500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:28.406463    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:28.406803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:28.406773    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:29.404622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:29.404578    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:29.405175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:29.404657    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:30.410857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:30.410819    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:30.411476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:30.410880    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:31.017327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.017220    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"]
Oct 30 18:48:31.017968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.017948    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0f15acf8-d3d8-47f9-b39f-73b172ca1d81" containerName="revoke-keys"
Oct 30 18:48:31.018073 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.017969    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f15acf8-d3d8-47f9-b39f-73b172ca1d81" containerName="revoke-keys"
Oct 30 18:48:31.018164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.018125    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f15acf8-d3d8-47f9-b39f-73b172ca1d81" containerName="revoke-keys"
Oct 30 18:48:31.024108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.023253    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.030831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.029885    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:48:31.030831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.030699    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"]
Oct 30 18:48:31.031449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.031423    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:48:31.142170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.137131    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"]
Oct 30 18:48:31.144585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.144551    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"]
Oct 30 18:48:31.144732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.144698    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.237671    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwzb\" (UniqueName: \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.238255    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrzt\" (UniqueName: \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.238323    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.238364    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.238440    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.245132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.238529    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.339501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339416    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrzt\" (UniqueName: \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.339656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339506    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.339656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339555    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.339656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339616    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.339761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339725    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.339798 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.339762    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwzb\" (UniqueName: \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.340612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.340583    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.341144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.341116    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.342768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.342729    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.342998 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.342981    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.352584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.352558    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwzb\" (UniqueName: \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\") pod \"maas-api-revoke-sub-89a0b435-48plv\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.352795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.352775    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrzt\" (UniqueName: \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\") pod \"maas-api-revoke-sub-95191225-vm22g\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.355820 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.355804    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:31.412973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.412479    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:31.412973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.412553    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:31.473267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.473236    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:31.725246 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:31.725208    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da829f1_d552_4728_af0d_543a90b19ae3.slice/crio-b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047 WatchSource:0}: Error finding container b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047: Status 404 returned error can't find the container with id b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047
Oct 30 18:48:31.731405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.731169    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"]
Oct 30 18:48:31.852437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:31.852410    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"]
Oct 30 18:48:32.364297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.364263    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 18:48:32.369032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.368997    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.391729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.391703    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 18:48:32.407159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.405057    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 18:48:32.407159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.405124    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 18:48:32.451831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.451764    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cptq\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.451850    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.451937    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.451998    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.452033    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.452078    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.452274    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.452310    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.452499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.452335    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.553982    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554014    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554033    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cptq\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554252    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554301    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554355    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554394    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554419    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.554632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554524    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.555041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554815    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.555041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554837    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.555041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.554859    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.555041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.555004    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.557135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.557075    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.557259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.557204    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.570951 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.570925    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.571165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.571146    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cptq\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.655256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.655217    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g" event={"ID":"bf729ee1-6872-4021-a9db-033f5aca60e1","Type":"ContainerStarted","Data":"916c1391308e3db925613bed99a67d39995268ce5d4844dfb54e297aad424a9c"}
Oct 30 18:48:32.655403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.655269    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g" event={"ID":"bf729ee1-6872-4021-a9db-033f5aca60e1","Type":"ContainerStarted","Data":"7b9827a6880377758d3de974cf908176006f24b96be4ff22c4229fa1c0f394da"}
Oct 30 18:48:32.657146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.657121    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv" event={"ID":"0da829f1-d552-4728-af0d-543a90b19ae3","Type":"ContainerStarted","Data":"efe4cab9bc8311499220fbe0a9bf315953fc283a770cf9fa623069c0352c6649"}
Oct 30 18:48:32.657272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.657185    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv" event={"ID":"0da829f1-d552-4728-af0d-543a90b19ae3","Type":"ContainerStarted","Data":"b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047"}
Oct 30 18:48:32.688100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.688035    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g" podStartSLOduration=1.688016218 podStartE2EDuration="1.688016218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:32.68414963 +0000 UTC m=+932.947614606" watchObservedRunningTime="2026-10-30 18:48:32.688016218 +0000 UTC m=+932.951481162"
Oct 30 18:48:32.692432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.692408    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:32.710753 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.710708    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv" podStartSLOduration=2.710693817 podStartE2EDuration="2.710693817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:32.704834748 +0000 UTC m=+932.968299689" watchObservedRunningTime="2026-10-30 18:48:32.710693817 +0000 UTC m=+932.974158764"
Oct 30 18:48:32.995895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:32.995044    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 18:48:33.001448 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:33.001414    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a57750_7509_4b7b_b609_0d161d72f037.slice/crio-05e7f2d8ac2aaae000f55bcab6d4f37eef1ab577f397d78d7065f65e1c6d8cad WatchSource:0}: Error finding container 05e7f2d8ac2aaae000f55bcab6d4f37eef1ab577f397d78d7065f65e1c6d8cad: Status 404 returned error can't find the container with id 05e7f2d8ac2aaae000f55bcab6d4f37eef1ab577f397d78d7065f65e1c6d8cad
Oct 30 18:48:33.004608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.004575    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:33.004711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.004644    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:33.004711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.004671    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:33.640620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.640427    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:33.641804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.641780    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:33.642347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.642305    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:48:33.668252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.668223    5065 generic.go:356] "Generic (PLEG): container finished" podID="bf729ee1-6872-4021-a9db-033f5aca60e1" containerID="916c1391308e3db925613bed99a67d39995268ce5d4844dfb54e297aad424a9c" exitCode=0
Oct 30 18:48:33.668415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.668258    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g" event={"ID":"bf729ee1-6872-4021-a9db-033f5aca60e1","Type":"ContainerDied","Data":"916c1391308e3db925613bed99a67d39995268ce5d4844dfb54e297aad424a9c"}
Oct 30 18:48:33.670243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.670216    5065 generic.go:356] "Generic (PLEG): container finished" podID="0da829f1-d552-4728-af0d-543a90b19ae3" containerID="efe4cab9bc8311499220fbe0a9bf315953fc283a770cf9fa623069c0352c6649" exitCode=0
Oct 30 18:48:33.670356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.670257    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv" event={"ID":"0da829f1-d552-4728-af0d-543a90b19ae3","Type":"ContainerDied","Data":"efe4cab9bc8311499220fbe0a9bf315953fc283a770cf9fa623069c0352c6649"}
Oct 30 18:48:33.672428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.672409    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" event={"ID":"c4a57750-7509-4b7b-b609-0d161d72f037","Type":"ContainerStarted","Data":"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"}
Oct 30 18:48:33.672527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.672442    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" event={"ID":"c4a57750-7509-4b7b-b609-0d161d72f037","Type":"ContainerStarted","Data":"05e7f2d8ac2aaae000f55bcab6d4f37eef1ab577f397d78d7065f65e1c6d8cad"}
Oct 30 18:48:33.694366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.694235    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:33.852120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:33.852044    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podStartSLOduration=1.852026035 podStartE2EDuration="1.852026035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:33.844252735 +0000 UTC m=+934.107717684" watchObservedRunningTime="2026-10-30 18:48:33.852026035 +0000 UTC m=+934.115490985"
Oct 30 18:48:34.737201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.733131    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:34.737201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.733183    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:34.845416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.845378    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:48:34.850965 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.850943    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.858433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.857822    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-58158b-openshift-default-dockercfg-fv2rs\""
Oct 30 18:48:34.936224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.929171    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945079    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945184    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945222    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4cz\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945259    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945361    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945687    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945732    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945761    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:34.945972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:34.945814    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.086623    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.086856    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.086905    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.086931    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.086995    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.087063    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.087124    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.087176    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4cz\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.087233    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.087743    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.088032    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.109781    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.110714    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.120967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.110970    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.570868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.570839    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:35.571804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.571766    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.574608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.574552    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.577851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.575708    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4cz\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.577851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.577264    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\") pod \"e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") " pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.637182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.632642    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:35.637182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.634776    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\") pod \"0da829f1-d552-4728-af0d-543a90b19ae3\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") "
Oct 30 18:48:35.637182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.634877    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\") pod \"bf729ee1-6872-4021-a9db-033f5aca60e1\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") "
Oct 30 18:48:35.637182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.635554    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca" pod "0da829f1-d552-4728-af0d-543a90b19ae3" (UID: "0da829f1-d552-4728-af0d-543a90b19ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:35.637182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.635660    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:35.638381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.638361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca" pod "bf729ee1-6872-4021-a9db-033f5aca60e1" (UID: "bf729ee1-6872-4021-a9db-033f5aca60e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:35.696114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.694919    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:35.696114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.695001    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:35.750114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.747580    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\" (UniqueName: \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\") pod \"0da829f1-d552-4728-af0d-543a90b19ae3\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") "
Oct 30 18:48:35.750114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.747655    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\") pod \"bf729ee1-6872-4021-a9db-033f5aca60e1\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") "
Oct 30 18:48:35.750114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.747816    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\" (UniqueName: \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\") pod \"bf729ee1-6872-4021-a9db-033f5aca60e1\" (UID: \"bf729ee1-6872-4021-a9db-033f5aca60e1\") "
Oct 30 18:48:35.750114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.748132    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\") pod \"0da829f1-d552-4728-af0d-543a90b19ae3\" (UID: \"0da829f1-d552-4728-af0d-543a90b19ae3\") "
Oct 30 18:48:35.763707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.763572    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca" pod "bf729ee1-6872-4021-a9db-033f5aca60e1" (UID: "bf729ee1-6872-4021-a9db-033f5aca60e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:35.769362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.769223    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca" pod "0da829f1-d552-4728-af0d-543a90b19ae3" (UID: "0da829f1-d552-4728-af0d-543a90b19ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:35.770783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.770712    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:35.770783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.770747    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da829f1-d552-4728-af0d-543a90b19ae3-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:35.770783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.770763    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf729ee1-6872-4021-a9db-033f5aca60e1-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:35.772568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.772528    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt" pod "bf729ee1-6872-4021-a9db-033f5aca60e1" (UID: "bf729ee1-6872-4021-a9db-033f5aca60e1"). InnerVolumeSpecName "kube-api-access-jzrzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:35.773543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.773367    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb" pod "0da829f1-d552-4728-af0d-543a90b19ae3" (UID: "0da829f1-d552-4728-af0d-543a90b19ae3"). InnerVolumeSpecName "kube-api-access-tlwzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:35.776342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.776075    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:35.817962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.817882    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g" event={"ID":"bf729ee1-6872-4021-a9db-033f5aca60e1","Type":"ContainerDied","Data":"7b9827a6880377758d3de974cf908176006f24b96be4ff22c4229fa1c0f394da"}
Oct 30 18:48:35.833130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.826791    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"
Oct 30 18:48:35.833396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.833359    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9827a6880377758d3de974cf908176006f24b96be4ff22c4229fa1c0f394da"
Oct 30 18:48:35.843004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.842976    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"
Oct 30 18:48:35.843154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.843138    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv" event={"ID":"0da829f1-d552-4728-af0d-543a90b19ae3","Type":"ContainerDied","Data":"b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047"}
Oct 30 18:48:35.843232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.843174    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46fa07fdb4c634690a754efed632e43bcff4887acb13bc2746905306bffa047"
Oct 30 18:48:35.871941 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.871917    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlwzb\" (UniqueName: \"kubernetes.io/projected/0da829f1-d552-4728-af0d-543a90b19ae3-kube-api-access-tlwzb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:35.871941 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:35.871940    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrzt\" (UniqueName: \"kubernetes.io/projected/bf729ee1-6872-4021-a9db-033f5aca60e1-kube-api-access-jzrzt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:36.211619 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.205261    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:48:36.213631 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:36.213599    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e77fa7_04d1_48c8_995d_f8137d457d22.slice/crio-72c7cf897bddc69d4c7c80862e140290c1fe7986134ba3a334043b71a4925d52 WatchSource:0}: Error finding container 72c7cf897bddc69d4c7c80862e140290c1fe7986134ba3a334043b71a4925d52: Status 404 returned error can't find the container with id 72c7cf897bddc69d4c7c80862e140290c1fe7986134ba3a334043b71a4925d52
Oct 30 18:48:36.217540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.217496    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:36.217623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.217581    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:36.217660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.217623    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:36.695065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.693468    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:36.695065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.693522    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:36.873552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.873514    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerStarted","Data":"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"}
Oct 30 18:48:36.874190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.874163    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerStarted","Data":"72c7cf897bddc69d4c7c80862e140290c1fe7986134ba3a334043b71a4925d52"}
Oct 30 18:48:36.929727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:36.929680    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podStartSLOduration=2.929666562 podStartE2EDuration="2.929666562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:36.902535471 +0000 UTC m=+937.166000421" watchObservedRunningTime="2026-10-30 18:48:36.929666562 +0000 UTC m=+937.193131509"
Oct 30 18:48:37.695111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:37.694465    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:37.695111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:37.694527    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:37.778466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:37.778432    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:37.815108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:37.814535    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:38.282705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.282673    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:38.287781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.287761    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:48:38.702166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.701355    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:38.715270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.715225    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:38.797111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.787351    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:38.842262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:38.842223    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.077112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.063328    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:48:39.138113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.137066    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:48:39.505677 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.505578    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.563236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.563180    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:39.623110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.623040    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.699806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.699772    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:39.700123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.700068    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:39.812111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.811485    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:48:39.871112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.869367    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:39.966569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:39.901573    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.066274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.066179    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.149277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.139175    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.328330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.328188    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.443265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.443224    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:40.443562 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.443512    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.522513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.522458    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.601602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.601506    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.650134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.631787    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676273    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676340    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676385    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676417    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676450    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:40.684121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.676481    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:40.816796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.816236    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.911659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.911569    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.917154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.912195    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:40.917154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:40.912238    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.017229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.011513    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:41.326123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.300503    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:41.689161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.180284    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.689161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.066973    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.689161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.365802    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:41.689161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.685604    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.721742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.721690    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.743503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.729679    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:41.743503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.729733    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:41.743503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.730007    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.769249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.744637    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.769249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.744895    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:41.787175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.786681    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:41.787175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.786742    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.919145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.881371    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:41.919145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.881434    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.978168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.977257    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.009114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:41.990210    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:42.105000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.104961    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.105302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.045142    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.667114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.658595    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.704115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.692233    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.704115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.701453    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:42.704115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.703427    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:42.733413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.731163    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.733413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.731524    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.747755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:42.745437    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.049169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.016255    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:48:43.111113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.110245    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.422165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.421886    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:48:43.732219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.667177    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.735220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.735171    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.835184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.751197    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.835184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.763674    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:48:43.835184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.764425    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:43.835184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.764463    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:43.870111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.853905    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.002415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:43.972468    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.021585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.021543    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.236762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.115320    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.619432    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"]
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620416    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bf729ee1-6872-4021-a9db-033f5aca60e1" containerName="revoke-keys"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620439    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf729ee1-6872-4021-a9db-033f5aca60e1" containerName="revoke-keys"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620502    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0da829f1-d552-4728-af0d-543a90b19ae3" containerName="revoke-keys"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620509    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da829f1-d552-4728-af0d-543a90b19ae3" containerName="revoke-keys"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620677    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf729ee1-6872-4021-a9db-033f5aca60e1" containerName="revoke-keys"
Oct 30 18:48:44.623108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.620703    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da829f1-d552-4728-af0d-543a90b19ae3" containerName="revoke-keys"
Oct 30 18:48:44.678126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.671283    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.747106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.747051    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:48:44.756583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.756333    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.763987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.763943    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.765565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.765535    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:48:44.769975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.768115    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:44.770954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:44.770925    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:45.033162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.014267    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"]
Oct 30 18:48:45.033162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.015869    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.065639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.065608    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:48:45.104995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.104957    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.147940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.147900    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.148318 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.148276    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.148597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.148577    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kj8\" (UniqueName: \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.187373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.187329    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.256580    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.280192    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.281164    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.422626    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.422792    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kj8\" (UniqueName: \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.429210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.425518    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.444684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.444404    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.445467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.445334    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:48:45.452206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.452180    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:48:45.496661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.496626    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.526990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.526956    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kj8\" (UniqueName: \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\") pod \"maas-api-revoke-sub-d062c581-spdl8\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.602457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.602426    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:48:45.650860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.650826    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:48:45.670334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.670311    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:48:45.697230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.697150    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:48:45.699107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.699057    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 18:48:45.801119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.801046    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:45.970970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.970937    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:48:45.989163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:45.983121    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.132138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.123899    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-58158b\""
Oct 30 18:48:46.132138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.128250    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-58158b-dockercfg-ggl2z\""
Oct 30 18:48:46.176111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.172535    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8swm\" (UniqueName: \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.176111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.172797    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.186113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.184959    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:48:46.278344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.277151    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.278344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.277311    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8swm\" (UniqueName: \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.281111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.278783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.319032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.318825    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8swm\" (UniqueName: \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\") pod \"payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:46.439550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.439507    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"]
Oct 30 18:48:46.599104 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:46.583289    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:47.098306 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:47.098218    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2081066_aefb_4fbb_b638_86b444a07d6b.slice/crio-505b8a5cf9ed7822d241538af3e9fac132ea787f749d6d6e6a10c27107890783 WatchSource:0}: Error finding container 505b8a5cf9ed7822d241538af3e9fac132ea787f749d6d6e6a10c27107890783: Status 404 returned error can't find the container with id 505b8a5cf9ed7822d241538af3e9fac132ea787f749d6d6e6a10c27107890783
Oct 30 18:48:47.099627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.099602    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:48:47.246401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.246367    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:47.247898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.247874    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:47.248716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.248693    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 18:48:47.294623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.294593    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" event={"ID":"e2081066-aefb-4fbb-b638-86b444a07d6b","Type":"ContainerStarted","Data":"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"}
Oct 30 18:48:47.294767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.294640    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" event={"ID":"e2081066-aefb-4fbb-b638-86b444a07d6b","Type":"ContainerStarted","Data":"505b8a5cf9ed7822d241538af3e9fac132ea787f749d6d6e6a10c27107890783"}
Oct 30 18:48:47.295362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.295334    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:47.297458 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.297429    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8" event={"ID":"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5","Type":"ContainerStarted","Data":"04f94974e194d2f3049395a7cfe72c02e5d35d8b23a19dd81706438502bd97c0"}
Oct 30 18:48:47.297585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.297486    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8" event={"ID":"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5","Type":"ContainerStarted","Data":"9020b6eb8e9cb5f465cbb820271129b318edd5bdc7142a8256f9e3965b55472e"}
Oct 30 18:48:47.417963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.417865    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8" podStartSLOduration=4.417848187 podStartE2EDuration="4.417848187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.349867041 +0000 UTC m=+947.613331988" watchObservedRunningTime="2026-10-30 18:48:47.417848187 +0000 UTC m=+947.681313134"
Oct 30 18:48:47.419944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.419895    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" podStartSLOduration=2.41987638 podStartE2EDuration="2.41987638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.413501995 +0000 UTC m=+947.676966957" watchObservedRunningTime="2026-10-30 18:48:47.41987638 +0000 UTC m=+947.683341331"
Oct 30 18:48:47.449003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.448971    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 18:48:47.453694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.453679    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.498782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.495263    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 18:48:47.551942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.551910    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.551956    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552010    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552064    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552132    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552165    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552210    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552248    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.552537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.552443    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bcm\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654122    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654203    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654234    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654264    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654309    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654344    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654506    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654573    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bcm\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654624    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654635    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654681    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.654882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.654827    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.655143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.655059    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.655253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.655056    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.656765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.656744    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.656998 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.656969    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.713354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.713256    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bcm\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.718131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.714253    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:47.769296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:47.769266    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:48.150368 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:48.150334    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caf1e42_14fd_432b_b3d6_164c7d02ec66.slice/crio-f91ebb438eefe09a2d81e2a0e4534b872dce8824461100fd9a59d8aceb567fce WatchSource:0}: Error finding container f91ebb438eefe09a2d81e2a0e4534b872dce8824461100fd9a59d8aceb567fce: Status 404 returned error can't find the container with id f91ebb438eefe09a2d81e2a0e4534b872dce8824461100fd9a59d8aceb567fce
Oct 30 18:48:48.153285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.153253    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 18:48:48.153285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.153264    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:48.153501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.153326    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:48.153501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.153353    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:48.314654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.314618    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" event={"ID":"1caf1e42-14fd-432b-b3d6-164c7d02ec66","Type":"ContainerStarted","Data":"79813a290fdebb9c8eb745ea26fdae4f5d8b80d8e7c5b4d6842cb7374b90e937"}
Oct 30 18:48:48.314887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.314682    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" event={"ID":"1caf1e42-14fd-432b-b3d6-164c7d02ec66","Type":"ContainerStarted","Data":"f91ebb438eefe09a2d81e2a0e4534b872dce8824461100fd9a59d8aceb567fce"}
Oct 30 18:48:48.317235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.317155    5065 generic.go:356] "Generic (PLEG): container finished" podID="0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" containerID="04f94974e194d2f3049395a7cfe72c02e5d35d8b23a19dd81706438502bd97c0" exitCode=0
Oct 30 18:48:48.318478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.318448    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8" event={"ID":"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5","Type":"ContainerDied","Data":"04f94974e194d2f3049395a7cfe72c02e5d35d8b23a19dd81706438502bd97c0"}
Oct 30 18:48:48.371730 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.371680    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podStartSLOduration=1.37165885 podStartE2EDuration="1.37165885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:48.36401116 +0000 UTC m=+948.627476145" watchObservedRunningTime="2026-10-30 18:48:48.37165885 +0000 UTC m=+948.635123798"
Oct 30 18:48:48.769368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:48.769335    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:49.562872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.562839    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:49.717570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.717493    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\" (UniqueName: \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\") pod \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") "
Oct 30 18:48:49.717766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.717604    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\") pod \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") "
Oct 30 18:48:49.717766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.717633    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\") pod \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\" (UID: \"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5\") "
Oct 30 18:48:49.718117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.718053    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca" pod "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" (UID: "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:49.718329 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.718311    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:49.720707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.720667    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca" pod "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" (UID: "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:49.720896 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.720875    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8" pod "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" (UID: "0fbfe700-0eee-4dcd-a1c0-862abd70a4a5"). InnerVolumeSpecName "kube-api-access-x8kj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:49.771315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.771292    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:49.771417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.771337    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:49.819525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.819477    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kj8\" (UniqueName: \"kubernetes.io/projected/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-kube-api-access-x8kj8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:49.819525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:49.819522    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:50.341132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:50.339985    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"
Oct 30 18:48:50.341132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:50.339989    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8" event={"ID":"0fbfe700-0eee-4dcd-a1c0-862abd70a4a5","Type":"ContainerDied","Data":"9020b6eb8e9cb5f465cbb820271129b318edd5bdc7142a8256f9e3965b55472e"}
Oct 30 18:48:50.341132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:50.340150    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9020b6eb8e9cb5f465cbb820271129b318edd5bdc7142a8256f9e3965b55472e"
Oct 30 18:48:50.769679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:50.769649    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:50.770144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:50.769707    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:51.770173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:51.770140    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:51.770552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:51.770206    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:52.769824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:52.769789    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:52.770003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:52.769846    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:53.060655 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:48:53.060560    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:48:53.769764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:53.769739    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:53.769986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:53.769965    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:54.769779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:54.769745    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 30 18:48:54.770236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:54.769806    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 30 18:48:55.801539 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:55.801502    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:55.802076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:55.802042    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:55.803132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:55.803113    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 18:48:57.791530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.791503    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:48:57.792276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.792254    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" containerName="revoke-keys"
Oct 30 18:48:57.792276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.792273    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" containerName="revoke-keys"
Oct 30 18:48:57.792486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.792409    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" containerName="revoke-keys"
Oct 30 18:48:57.797599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.797580    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.808703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.808679    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f44051-openshift-default-dockercfg-5wcgd\""
Oct 30 18:48:57.825703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.825677    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:48:57.948171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.948062    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hgj\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.948540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.948509    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.948792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.948762    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.948981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.948951    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.949188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.949158    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.949452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.949419    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.949659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.949640    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.949751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.949735    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:57.949823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:57.949766    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.054025 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.053931    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.054796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.054776    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.055475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.055454    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060270    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.054647    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.057942    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060531    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hgj\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060589    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060641    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060690    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.060816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.060757    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.061734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.061220    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.061734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.061564    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.061734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.061687    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.066617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.066594    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.091109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.081311    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.134246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.134146    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.149653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.143790    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hgj\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:58.421162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:58.420340    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:48:59.046028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.045984    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:48:59.052248 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:48:59.047339    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458092dc_c9a1_4747_a2ec_56fa503b1c49.slice/crio-25b1a7f614de3bf6cfce7c5699987da37f08167054f8f794eef11cd8d38e99f9 WatchSource:0}: Error finding container 25b1a7f614de3bf6cfce7c5699987da37f08167054f8f794eef11cd8d38e99f9: Status 404 returned error can't find the container with id 25b1a7f614de3bf6cfce7c5699987da37f08167054f8f794eef11cd8d38e99f9
Oct 30 18:48:59.052248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.050693    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:59.052248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.050781    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:59.052248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.050822    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:48:59.333467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.333377    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:48:59.498667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.498627    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerStarted","Data":"83f122f10a6f7080f06b49c72e1a0603861246cb2bcc560a259c721660f45b73"}
Oct 30 18:48:59.498837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.498692    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerStarted","Data":"25b1a7f614de3bf6cfce7c5699987da37f08167054f8f794eef11cd8d38e99f9"}
Oct 30 18:48:59.560234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:48:59.560190    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podStartSLOduration=2.560175773 podStartE2EDuration="2.560175773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:59.556420774 +0000 UTC m=+959.819885723" watchObservedRunningTime="2026-10-30 18:48:59.560175773 +0000 UTC m=+959.823640723"
Oct 30 18:49:00.423347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:00.423313    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:00.429544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:00.429516    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:00.509027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:00.509000    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:00.509829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:00.509810    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:03.267056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.267017    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:49:03.274145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.274123    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.287682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.287652    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f44051-dockercfg-4z9rc\""
Oct 30 18:49:03.324016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.323981    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:49:03.347395 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.347359    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f44051\""
Oct 30 18:49:03.615257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.597758    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5qq\" (UniqueName: \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.615597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.615561    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.720342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.720307    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5qq\" (UniqueName: \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.720663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.720637    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.721773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.721748    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.790793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.790761    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5qq\" (UniqueName: \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\") pod \"payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:03.898267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:03.898183    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:04.478189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:04.466180    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:49:04.908219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:04.845535    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" event={"ID":"ef8a87b0-8479-4b99-b6e5-43438ff2e45e","Type":"ContainerStarted","Data":"7679062b3fdb22d009b391012704a8a5fcdbfd63e8fe8188387a7cc3ea2b568a"}
Oct 30 18:49:05.966676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:05.966639    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" event={"ID":"ef8a87b0-8479-4b99-b6e5-43438ff2e45e","Type":"ContainerStarted","Data":"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"}
Oct 30 18:49:05.968262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:05.968240    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:06.052606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:06.052541    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" podStartSLOduration=3.052521852 podStartE2EDuration="3.052521852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:06.037754938 +0000 UTC m=+966.301219885" watchObservedRunningTime="2026-10-30 18:49:06.052521852 +0000 UTC m=+966.315986802"
Oct 30 18:49:06.735110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:06.734228    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:06.739111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:06.735374    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:06.752109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:06.751740    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.021705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:07.004013    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:49:07.035303 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:07.025153    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.097997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:07.097959    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:16.292393 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.292340    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:49:16.345330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.344773    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:49:16.345330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.344862    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.445466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.445355    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f9521e-openshift-default-dockercfg-zrpvc\""
Oct 30 18:49:16.467037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.466929    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.467382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.467360    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.481470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.481381    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.481784 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.481763    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.482264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.482237    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.483057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.483033    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.483289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.483253    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.500236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.500207    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.500511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.500472    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdf8d\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.506017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.505990    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerDied","Data":"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"}
Oct 30 18:49:16.527109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.507106    5065 scope.go:122] "RemoveContainer" containerID="650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"
Oct 30 18:49:16.530554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.530518    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:16.530763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.530741    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:16.530961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.530932    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:16.532169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.532148    5065 generic.go:356] "Generic (PLEG): container finished" podID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerID="650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b" exitCode=0
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.605770    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.605847    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.605874    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.605941    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606118    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606151    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606163    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606235    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606279    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdf8d\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.606375    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.607971    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.609297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.618134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.609960    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.647647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.641568    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.647647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.642313    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.647647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.644891    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.647647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.645149    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdf8d\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\") pod \"e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") " pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:16.670109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:16.662942    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:17.007453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.007415    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:17.007571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.007504    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:17.007571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.007555    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:17.015953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.015920    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:49:17.548420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.548385    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerStarted","Data":"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"}
Oct 30 18:49:17.553360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.553334    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerStarted","Data":"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"}
Oct 30 18:49:17.553497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.553385    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerStarted","Data":"762be95533e4d6a58597c9fd7e3d133829328742b6a3ba26bea9224956cb88fa"}
Oct 30 18:49:17.664129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:17.663924    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:18.205305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.205274    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:49:18.245315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.245264    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podStartSLOduration=2.245248219 podStartE2EDuration="2.245248219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:17.622491971 +0000 UTC m=+977.885956931" watchObservedRunningTime="2026-10-30 18:49:18.245248219 +0000 UTC m=+978.508713166"
Oct 30 18:49:18.470219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.467505    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:49:18.470219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.467625    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:18.470219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.467678    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:18.670479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:18.670452    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:19.467497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:19.467453    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:19.467690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:19.467511    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:19.606111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:19.605509    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:19.616863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:19.612318    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:49:20.467326 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:20.467288    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:20.467729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:20.467352    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:21.467491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.467447    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:21.467994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.467527    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:21.638705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.638665    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:49:21.643275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.643258    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:21.676299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.676279    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9521e\""
Oct 30 18:49:21.731183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.730461    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:49:21.768549 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.768515    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98kb\" (UniqueName: \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:21.768714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.768616    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.007162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.992702    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.007162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.993100    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98kb\" (UniqueName: \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.007162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:21.994362    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.166576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.166538    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98kb\" (UniqueName: \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\") pod \"payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.258833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.258801    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:22.473593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.468471    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:22.473593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.468526    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:22.724985 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:49:22.724950    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8719b6b_f2b5_433c_a68a_f310fef9838a.slice/crio-a89d594cb6d3097309d0f12f66849d9efa7c15e69b3220afbc7f3a72c804f20a WatchSource:0}: Error finding container a89d594cb6d3097309d0f12f66849d9efa7c15e69b3220afbc7f3a72c804f20a: Status 404 returned error can't find the container with id a89d594cb6d3097309d0f12f66849d9efa7c15e69b3220afbc7f3a72c804f20a
Oct 30 18:49:22.727548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.727526    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:49:22.971060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.971019    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:49:22.991327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:22.976311    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.030202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.028679    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-30970e\""
Oct 30 18:49:23.030202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.028828    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-30970e-dockercfg-fwgzg\""
Oct 30 18:49:23.081122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.071140    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:49:23.227108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.217374    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48gc\" (UniqueName: \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.227108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.217612    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.252169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.238424    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:49:23.354111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.353620    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.354111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.353912    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48gc\" (UniqueName: \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.382150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.355270    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.388139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.386035    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:49:23.388139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.386196    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.430874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.427943    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48gc\" (UniqueName: \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\") pod \"payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.431468 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:49:23.431423    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:49:23.455165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.455140    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.455386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.455369    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjq6\" (UniqueName: \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.467728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.467702    5065 patch_prober.go:29] interesting pod/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 18:49:23.467802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.467760    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 18:49:23.569465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.569425    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjq6\" (UniqueName: \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.575636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.575545    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.588167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.577276    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.598510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.598458    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjq6\" (UniqueName: \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\") pod \"payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.609982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.609948    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:23.661426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.661386    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" event={"ID":"d8719b6b-f2b5-433c-a68a-f310fef9838a","Type":"ContainerStarted","Data":"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"}
Oct 30 18:49:23.661587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.661443    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" event={"ID":"d8719b6b-f2b5-433c-a68a-f310fef9838a","Type":"ContainerStarted","Data":"a89d594cb6d3097309d0f12f66849d9efa7c15e69b3220afbc7f3a72c804f20a"}
Oct 30 18:49:23.662393 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.662370    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:23.697434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.696925    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" podStartSLOduration=2.696904582 podStartE2EDuration="2.696904582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:23.687069095 +0000 UTC m=+983.950534045" watchObservedRunningTime="2026-10-30 18:49:23.696904582 +0000 UTC m=+983.960369531"
Oct 30 18:49:23.752115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.752073    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:23.945604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:23.945543    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:49:24.175016 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:49:24.174985    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd665943_9d99_46ac_b5f4_15c785bb39f2.slice/crio-ce54ed77a93407078181b92718fad3e91c38aaba6b052ced0b0a75a67ceefd8b WatchSource:0}: Error finding container ce54ed77a93407078181b92718fad3e91c38aaba6b052ced0b0a75a67ceefd8b: Status 404 returned error can't find the container with id ce54ed77a93407078181b92718fad3e91c38aaba6b052ced0b0a75a67ceefd8b
Oct 30 18:49:24.176149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.176121    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:49:24.481057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.480379    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:49:24.483821 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.483707    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:49:24.484934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.484913    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:49:24.693320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.693282    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" event={"ID":"bd665943-9d99-46ac-b5f4-15c785bb39f2","Type":"ContainerStarted","Data":"52428685764b874f4907873975228bec59af11b417ad4e816fe5fc1a02cb81ea"}
Oct 30 18:49:24.693797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.693349    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" event={"ID":"bd665943-9d99-46ac-b5f4-15c785bb39f2","Type":"ContainerStarted","Data":"ce54ed77a93407078181b92718fad3e91c38aaba6b052ced0b0a75a67ceefd8b"}
Oct 30 18:49:24.693797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.693380    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:24.695396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.695360    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" event={"ID":"23357381-db62-4c25-a29a-8305dfca964b","Type":"ContainerStarted","Data":"38b4593cc811079b005f2f7b084a69e9dde1219aea7a60d1987aee9173be24d1"}
Oct 30 18:49:24.695540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.695456    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:24.695540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.695481    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" event={"ID":"23357381-db62-4c25-a29a-8305dfca964b","Type":"ContainerStarted","Data":"7011bfbd1c708a4d3a175cf8a9cc00c9f81313d3c0b41304211a54b7ce79f175"}
Oct 30 18:49:24.724248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.724201    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" podStartSLOduration=2.724186194 podStartE2EDuration="2.724186194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:24.723234493 +0000 UTC m=+984.986699480" watchObservedRunningTime="2026-10-30 18:49:24.724186194 +0000 UTC m=+984.987651141"
Oct 30 18:49:24.749758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:24.749668    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" podStartSLOduration=2.749651137 podStartE2EDuration="2.749651137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:24.744788757 +0000 UTC m=+985.008253722" watchObservedRunningTime="2026-10-30 18:49:24.749651137 +0000 UTC m=+985.013116085"
Oct 30 18:49:28.868301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.868262    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"]
Oct 30 18:49:28.877555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.876693    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:28.885435 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.885406    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"]
Oct 30 18:49:28.885993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.885848    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:49:28.888490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.888469    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:49:28.990496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.990462    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:28.990699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.990672    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp22h\" (UniqueName: \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:28.990775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:28.990738    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.092694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.092659    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp22h\" (UniqueName: \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.092893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.092728    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.092979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.092962    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.093672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.093636    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.095796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.095767    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.107154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.107129    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp22h\" (UniqueName: \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\") pod \"maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.238007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.237935    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:29.706913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.694553    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"]
Oct 30 18:49:29.784536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:29.784491    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt" event={"ID":"44df998a-6131-49e0-aa17-be4e7935eeb0","Type":"ContainerStarted","Data":"454ccd734ec96f250450120aba562dba8636128eb6c3704ab078c845d547f870"}
Oct 30 18:49:30.822588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:30.818244    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt" event={"ID":"44df998a-6131-49e0-aa17-be4e7935eeb0","Type":"ContainerStarted","Data":"90fda4db04b9fa5473520955f14e286daa46ccc1a517ce62b821d3805701364a"}
Oct 30 18:49:32.043926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:32.043889    5065 generic.go:356] "Generic (PLEG): container finished" podID="44df998a-6131-49e0-aa17-be4e7935eeb0" containerID="90fda4db04b9fa5473520955f14e286daa46ccc1a517ce62b821d3805701364a" exitCode=0
Oct 30 18:49:32.044882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:32.044855    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt" event={"ID":"44df998a-6131-49e0-aa17-be4e7935eeb0","Type":"ContainerDied","Data":"90fda4db04b9fa5473520955f14e286daa46ccc1a517ce62b821d3805701364a"}
Oct 30 18:49:33.552234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.551882    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:33.555544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.555515    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\") pod \"44df998a-6131-49e0-aa17-be4e7935eeb0\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") "
Oct 30 18:49:33.556242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.555719    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\") pod \"44df998a-6131-49e0-aa17-be4e7935eeb0\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") "
Oct 30 18:49:33.556242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.555921    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\" (UniqueName: \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\") pod \"44df998a-6131-49e0-aa17-be4e7935eeb0\" (UID: \"44df998a-6131-49e0-aa17-be4e7935eeb0\") "
Oct 30 18:49:33.556600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.556552    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca" pod "44df998a-6131-49e0-aa17-be4e7935eeb0" (UID: "44df998a-6131-49e0-aa17-be4e7935eeb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:33.561021 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.560991    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca" pod "44df998a-6131-49e0-aa17-be4e7935eeb0" (UID: "44df998a-6131-49e0-aa17-be4e7935eeb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:33.561342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.561313    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h" pod "44df998a-6131-49e0-aa17-be4e7935eeb0" (UID: "44df998a-6131-49e0-aa17-be4e7935eeb0"). InnerVolumeSpecName "kube-api-access-qp22h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:33.658161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.658121    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp22h\" (UniqueName: \"kubernetes.io/projected/44df998a-6131-49e0-aa17-be4e7935eeb0-kube-api-access-qp22h\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:33.658161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.658163    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:33.658423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:33.658180    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44df998a-6131-49e0-aa17-be4e7935eeb0-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:34.280006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:34.278203    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"
Oct 30 18:49:34.280006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:34.278239    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt" event={"ID":"44df998a-6131-49e0-aa17-be4e7935eeb0","Type":"ContainerDied","Data":"454ccd734ec96f250450120aba562dba8636128eb6c3704ab078c845d547f870"}
Oct 30 18:49:34.280006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:34.278284    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454ccd734ec96f250450120aba562dba8636128eb6c3704ab078c845d547f870"
Oct 30 18:49:35.737112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:35.730491    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:49:35.737112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:35.731231    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:49:35.737112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:35.733682    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:49:39.281684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:39.281258    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:49:39.281684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:39.281340    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:42.907527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.907459    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:49:42.908239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.908215    5065 cpu_manager.go:414] "containerMap: removing container" podUID="44df998a-6131-49e0-aa17-be4e7935eeb0" containerName="revoke-keys"
Oct 30 18:49:42.908239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.908233    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44df998a-6131-49e0-aa17-be4e7935eeb0" containerName="revoke-keys"
Oct 30 18:49:42.908383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.908353    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="44df998a-6131-49e0-aa17-be4e7935eeb0" containerName="revoke-keys"
Oct 30 18:49:42.911661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.911637    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:42.931301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:42.931278    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:49:43.017948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.017902    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018013    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018049    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018065    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018138    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018222    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018259    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018589    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbdv\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.018877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.018853    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.142014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.141977    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbdv\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.143140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.143075    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.143982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.143962    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.144557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.144538    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.144966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.144947    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.145131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.145111    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.145279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.145263    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.145407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.145392    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.145537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.145521    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.146013 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.145982    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.146120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.144463    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.146995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.146948    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.147720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.147642    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.148035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.147907    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.177403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.166065    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.177403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.176032    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbdv\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.177403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.176561    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.177403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.177312    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.243243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.243216    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:43.526501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.526451    5065 generic.go:356] "Generic (PLEG): container finished" podID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerID="83f122f10a6f7080f06b49c72e1a0603861246cb2bcc560a259c721660f45b73" exitCode=0
Oct 30 18:49:43.526664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.526499    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerDied","Data":"83f122f10a6f7080f06b49c72e1a0603861246cb2bcc560a259c721660f45b73"}
Oct 30 18:49:43.527009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.526992    5065 scope.go:122] "RemoveContainer" containerID="83f122f10a6f7080f06b49c72e1a0603861246cb2bcc560a259c721660f45b73"
Oct 30 18:49:43.528424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528404    5065 generic.go:356] "Generic (PLEG): container finished" podID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerID="17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444" exitCode=0
Oct 30 18:49:43.528508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528486    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerDied","Data":"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"}
Oct 30 18:49:43.528694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528667    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.528769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528753    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.528815 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528802    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.528899 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.528885    5065 scope.go:122] "RemoveContainer" containerID="17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"
Oct 30 18:49:43.530230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.530207    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.530297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.530264    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.530330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.530300    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.690322 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:49:43.690288    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5266cc25_e43e_4ad1_b80f_2be71aec35d7.slice/crio-3bd8665bd8c57370eb471883792001dbc8d3843405267a2c6823fee365b6538b WatchSource:0}: Error finding container 3bd8665bd8c57370eb471883792001dbc8d3843405267a2c6823fee365b6538b: Status 404 returned error can't find the container with id 3bd8665bd8c57370eb471883792001dbc8d3843405267a2c6823fee365b6538b
Oct 30 18:49:43.699612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.699486    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:49:43.706776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.702830    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.706776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.702910    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:43.706776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:43.702958    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:49:44.556370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.556329    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" event={"ID":"5266cc25-e43e-4ad1-b80f-2be71aec35d7","Type":"ContainerStarted","Data":"d90866b060c0c9b6fedba5997e701f46b433c329cd4c9abc82cd0434fc28c7aa"}
Oct 30 18:49:44.556955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.556922    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" event={"ID":"5266cc25-e43e-4ad1-b80f-2be71aec35d7","Type":"ContainerStarted","Data":"3bd8665bd8c57370eb471883792001dbc8d3843405267a2c6823fee365b6538b"}
Oct 30 18:49:44.559231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.559207    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerStarted","Data":"7f08430e5f7107f047028f44860e030feb534544619f235a85c045765240ccfa"}
Oct 30 18:49:44.561701 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.561671    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerStarted","Data":"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"}
Oct 30 18:49:44.616113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.615799    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podStartSLOduration=2.615777618 podStartE2EDuration="2.615777618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:44.599548142 +0000 UTC m=+1004.863013110" watchObservedRunningTime="2026-10-30 18:49:44.615777618 +0000 UTC m=+1004.879242568"
Oct 30 18:49:44.787113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.778505    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:49:44.828177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.802197    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:44.828177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:44.802261    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:45.249165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.246512    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:45.249165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.246657    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:45.249165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.246701    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:45.484201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.474394    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:45.484201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.474463    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:45.484201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.474980    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:45.511173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.509193    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:45.771189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.765696    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:45.776612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.776280    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:45.779417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.779373    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:45.779631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.779606    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:45.789145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:45.788156    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:46.252749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.252667    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:46.253144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.253059    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:46.424503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.424466    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:46.424773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.424743    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:46.779905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.779154    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:46.779905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:46.779256    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:47.248170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.247456    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:47.248170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.247519    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:47.431112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.430574    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:47.431112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.430634    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:47.781917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.781433    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:47.781917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:47.781496    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:48.246169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.245899    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:48.246169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.245973    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:48.431671 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.424926    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:48.431671 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.424994    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:48.612171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.610217    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"]
Oct 30 18:49:48.638115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.629303    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:48.716161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.715382    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:49:48.716161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.715478    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:49:48.869262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.837439    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:48.869262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.837501    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:48.947112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.920944    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"]
Oct 30 18:49:48.992152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.982231    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkph\" (UniqueName: \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:48.992152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.982308    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:48.992152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:48.982397    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.092172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.087607    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.092172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.088514    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkph\" (UniqueName: \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.092172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.088557    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.092172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.089742    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.112845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.112815    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.133190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.123437    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkph\" (UniqueName: \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\") pod \"maas-api-revoke-keys-e2e-api-a-58158b-62vn7\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.151459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.151427    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:49.255954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.255485    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:49.255954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.255558    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:49.438932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.438428    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:49.438932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.438487    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:49.822110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.796454    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:49.822110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:49.796511    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:50.296112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.281427    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:50.296112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.281482    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:50.869071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.869034    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:49:50.869355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.869329    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 18:49:50.875111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.875054    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"]
Oct 30 18:49:50.876209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.876170    5065 patch_prober.go:29] interesting pod/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:49:50.876523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:50.876390    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 18:49:51.201174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.201071    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7" event={"ID":"98693852-1b9e-4647-8603-f9a84aafe4ca","Type":"ContainerStarted","Data":"32d6834728bfbd29dabe167a88883e8e9d9e843ff29831176fb143d8a0d4bf55"}
Oct 30 18:49:51.255898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.254391    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:49:51.255898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.254451    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 18:49:51.429193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.429076    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:51.430838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.430813    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:49:51.783833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.783798    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:49:51.784269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.784251    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:49:51.785064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:51.785044    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:49:52.216579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.216542    5065 generic.go:356] "Generic (PLEG): container finished" podID="98693852-1b9e-4647-8603-f9a84aafe4ca" containerID="9418d8cee1ca4bb657ffe372ae2625ca909c4c9621874592ab031750bbf438f6" exitCode=0
Oct 30 18:49:52.216724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.216631    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7" event={"ID":"98693852-1b9e-4647-8603-f9a84aafe4ca","Type":"ContainerDied","Data":"9418d8cee1ca4bb657ffe372ae2625ca909c4c9621874592ab031750bbf438f6"}
Oct 30 18:49:52.319927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.315799    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:52.319927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.317533    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:52.319927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.318858    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:49:52.436478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.436408    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"]
Oct 30 18:49:52.441321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.441299    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.454158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.454119    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"]
Oct 30 18:49:52.499167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.495950    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.499167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.496398    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqt5\" (UniqueName: \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.499167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.496869    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.598672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.598641    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqt5\" (UniqueName: \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.598880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.598861    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.598978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.598961    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.599555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.599528    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.601527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.601507    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.612412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.612389    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqt5\" (UniqueName: \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\") pod \"maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:52.759751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:52.759734    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:53.055183 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:49:53.055156    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b02b5fa_bbe9_4e7f_a300_ad5777189c87.slice/crio-84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187 WatchSource:0}: Error finding container 84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187: Status 404 returned error can't find the container with id 84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187
Oct 30 18:49:53.059253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.059142    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"]
Oct 30 18:49:53.068061 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:49:53.068029    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:49:53.239240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.231464    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk" event={"ID":"6b02b5fa-bbe9-4e7f-a300-ad5777189c87","Type":"ContainerStarted","Data":"84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187"}
Oct 30 18:49:53.510389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.509407    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"]
Oct 30 18:49:53.524698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.516529    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4d94b-wxrrt"]
Oct 30 18:49:53.780161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.770717    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:53.830692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.830650    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\") pod \"98693852-1b9e-4647-8603-f9a84aafe4ca\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") "
Oct 30 18:49:53.830871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.830820    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\") pod \"98693852-1b9e-4647-8603-f9a84aafe4ca\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") "
Oct 30 18:49:53.830871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.830864    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\" (UniqueName: \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\") pod \"98693852-1b9e-4647-8603-f9a84aafe4ca\" (UID: \"98693852-1b9e-4647-8603-f9a84aafe4ca\") "
Oct 30 18:49:53.831994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.831962    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca" pod "98693852-1b9e-4647-8603-f9a84aafe4ca" (UID: "98693852-1b9e-4647-8603-f9a84aafe4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:53.834765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.834584    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph" pod "98693852-1b9e-4647-8603-f9a84aafe4ca" (UID: "98693852-1b9e-4647-8603-f9a84aafe4ca"). InnerVolumeSpecName "kube-api-access-dnkph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:53.835623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.835599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca" pod "98693852-1b9e-4647-8603-f9a84aafe4ca" (UID: "98693852-1b9e-4647-8603-f9a84aafe4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:53.895860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.895831    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44df998a-6131-49e0-aa17-be4e7935eeb0" path="/var/lib/kubelet/pods/44df998a-6131-49e0-aa17-be4e7935eeb0/volumes"
Oct 30 18:49:53.932536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.932468    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:53.932536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.932500    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98693852-1b9e-4647-8603-f9a84aafe4ca-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:53.932536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:53.932515    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkph\" (UniqueName: \"kubernetes.io/projected/98693852-1b9e-4647-8603-f9a84aafe4ca-kube-api-access-dnkph\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:54.241263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:54.241237    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"
Oct 30 18:49:54.241263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:54.241252    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7" event={"ID":"98693852-1b9e-4647-8603-f9a84aafe4ca","Type":"ContainerDied","Data":"32d6834728bfbd29dabe167a88883e8e9d9e843ff29831176fb143d8a0d4bf55"}
Oct 30 18:49:54.241474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:54.241302    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32d6834728bfbd29dabe167a88883e8e9d9e843ff29831176fb143d8a0d4bf55"
Oct 30 18:49:54.243767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:54.243739    5065 generic.go:356] "Generic (PLEG): container finished" podID="6b02b5fa-bbe9-4e7f-a300-ad5777189c87" containerID="d2723feb20bf4fd0caf6a11dd70a48bf1893ec42f25255b1a07466f7adcea5fe" exitCode=0
Oct 30 18:49:54.243889 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:54.243839    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk" event={"ID":"6b02b5fa-bbe9-4e7f-a300-ad5777189c87","Type":"ContainerDied","Data":"d2723feb20bf4fd0caf6a11dd70a48bf1893ec42f25255b1a07466f7adcea5fe"}
Oct 30 18:49:55.445561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.445538    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:55.563674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.562445    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\") pod \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") "
Oct 30 18:49:55.563674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.562528    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\" (UniqueName: \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\") pod \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") "
Oct 30 18:49:55.563674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.562730    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\") pod \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\" (UID: \"6b02b5fa-bbe9-4e7f-a300-ad5777189c87\") "
Oct 30 18:49:55.568030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.567363    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca" pod "6b02b5fa-bbe9-4e7f-a300-ad5777189c87" (UID: "6b02b5fa-bbe9-4e7f-a300-ad5777189c87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:55.572480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.570104    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca" pod "6b02b5fa-bbe9-4e7f-a300-ad5777189c87" (UID: "6b02b5fa-bbe9-4e7f-a300-ad5777189c87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:55.581600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.581566    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5" pod "6b02b5fa-bbe9-4e7f-a300-ad5777189c87" (UID: "6b02b5fa-bbe9-4e7f-a300-ad5777189c87"). InnerVolumeSpecName "kube-api-access-8rqt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:55.664626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.664592    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:55.664626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.664617    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqt5\" (UniqueName: \"kubernetes.io/projected/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-kube-api-access-8rqt5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:55.664626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:55.664633    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b02b5fa-bbe9-4e7f-a300-ad5777189c87-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:56.265049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:56.265022    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"
Oct 30 18:49:56.265254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:56.265057    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk" event={"ID":"6b02b5fa-bbe9-4e7f-a300-ad5777189c87","Type":"ContainerDied","Data":"84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187"}
Oct 30 18:49:56.265254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:56.265124    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f9f077957cc4872ebc1a962e69a813deae78f6ceafebb5c4ded775cd7a8187"
Oct 30 18:49:58.839290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:58.830616    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:49:59.233174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:59.232508    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"]
Oct 30 18:49:59.369329 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:49:59.369258    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800848d5-phxnq"]
Oct 30 18:50:00.177397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:00.177335    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11e4613-303c-4f06-9800-f6c86e7b61fd" path="/var/lib/kubelet/pods/b11e4613-303c-4f06-9800-f6c86e7b61fd/volumes"
Oct 30 18:50:00.178377 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:00.178350    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"]
Oct 30 18:50:00.178582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:00.178568    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47df19d8-kn86m"]
Oct 30 18:50:01.449345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.449311    5065 scope.go:122] "RemoveContainer" containerID="12d4fc5cc14bbd92f54ffa6ee93b4cf5e92f949c22459308fc88f8bd82df7f12"
Oct 30 18:50:01.468992 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.468956    5065 scope.go:122] "RemoveContainer" containerID="d0a11033470d8ce7fa0d3f8a07d239aa8ca78be11e932135cd5460c95c06bc94"
Oct 30 18:50:01.512440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.512411    5065 scope.go:122] "RemoveContainer" containerID="04c06eed4aa74058af403a3285589a025645c5a7ad6e534dddffe92fc016e05d"
Oct 30 18:50:01.528415 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.528390    5065 scope.go:122] "RemoveContainer" containerID="93ebffc1a56f5ca1162f17601040d59a78cff92b0094efd7d2f58f46252706f5"
Oct 30 18:50:01.543924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.543898    5065 scope.go:122] "RemoveContainer" containerID="c051e5a08db4cfc7b64c4f3d7ccee5bfc78c4175be557dc8cb260bf0b12918da"
Oct 30 18:50:01.735934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.735831    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:50:01.736219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:01.736107    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy" containerID="cri-o://a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74" gracePeriod=30
Oct 30 18:50:02.198108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.197142    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3759e6-80d4-4d96-bd03-1b7079099af3" path="/var/lib/kubelet/pods/3c3759e6-80d4-4d96-bd03-1b7079099af3/volumes"
Oct 30 18:50:02.249500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.249464    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"]
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.250719    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6b02b5fa-bbe9-4e7f-a300-ad5777189c87" containerName="revoke-keys"
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.250747    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b02b5fa-bbe9-4e7f-a300-ad5777189c87" containerName="revoke-keys"
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.250775    5065 cpu_manager.go:414] "containerMap: removing container" podUID="98693852-1b9e-4647-8603-f9a84aafe4ca" containerName="revoke-keys"
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.250782    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98693852-1b9e-4647-8603-f9a84aafe4ca" containerName="revoke-keys"
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.251025    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b02b5fa-bbe9-4e7f-a300-ad5777189c87" containerName="revoke-keys"
Oct 30 18:50:02.251443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.251049    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="98693852-1b9e-4647-8603-f9a84aafe4ca" containerName="revoke-keys"
Oct 30 18:50:02.266161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.265893    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"]
Oct 30 18:50:02.266161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.266022    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.350849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.350800    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2pr\" (UniqueName: \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.351017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.350982    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.351290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.351270    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.358321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.358287    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:02.358499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.358360    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:50:02.452982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.452893    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.454031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.452993    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2pr\" (UniqueName: \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.454031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.453046    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.454031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.453728    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.455923 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.455895    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.468503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.468466    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2pr\" (UniqueName: \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\") pod \"maas-api-revoke-keys-e2e-api-b-f44051-rpp46\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:02.621153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:02.612370    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:03.366316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:03.352481    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"]
Oct 30 18:50:03.567353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:03.567301    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46" event={"ID":"706d972e-87f6-4e69-b5b8-e085a5fd2f23","Type":"ContainerStarted","Data":"c76d71d9a427f067ac53a8729b559ac52e262031e39323783f169c556b60608d"}
Oct 30 18:50:03.831393 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:03.831328    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" podUID="e2081066-aefb-4fbb-b638-86b444a07d6b" containerName="payload-processing" containerID="cri-o://3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e" gracePeriod=25
Oct 30 18:50:04.227036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.227004    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:50:04.332160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.327195    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\") pod \"e2081066-aefb-4fbb-b638-86b444a07d6b\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") "
Oct 30 18:50:04.332160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.327280    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\" (UniqueName: \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\") pod \"e2081066-aefb-4fbb-b638-86b444a07d6b\" (UID: \"e2081066-aefb-4fbb-b638-86b444a07d6b\") "
Oct 30 18:50:04.357065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.354338    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume" pod "e2081066-aefb-4fbb-b638-86b444a07d6b" (UID: "e2081066-aefb-4fbb-b638-86b444a07d6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:04.361351 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.361315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm" pod "e2081066-aefb-4fbb-b638-86b444a07d6b" (UID: "e2081066-aefb-4fbb-b638-86b444a07d6b"). InnerVolumeSpecName "kube-api-access-t8swm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:04.429985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.429953    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2081066-aefb-4fbb-b638-86b444a07d6b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:04.429985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.429985    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8swm\" (UniqueName: \"kubernetes.io/projected/e2081066-aefb-4fbb-b638-86b444a07d6b-kube-api-access-t8swm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:04.577852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.577808    5065 generic.go:356] "Generic (PLEG): container finished" podID="e2081066-aefb-4fbb-b638-86b444a07d6b" containerID="3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e" exitCode=0
Oct 30 18:50:04.578299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.577860    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"
Oct 30 18:50:04.578299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.577868    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" event={"ID":"e2081066-aefb-4fbb-b638-86b444a07d6b","Type":"ContainerDied","Data":"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"}
Oct 30 18:50:04.578299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.577939    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx" event={"ID":"e2081066-aefb-4fbb-b638-86b444a07d6b","Type":"ContainerDied","Data":"505b8a5cf9ed7822d241538af3e9fac132ea787f749d6d6e6a10c27107890783"}
Oct 30 18:50:04.578299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.577966    5065 scope.go:122] "RemoveContainer" containerID="3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"
Oct 30 18:50:04.609125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.608072    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46" event={"ID":"706d972e-87f6-4e69-b5b8-e085a5fd2f23","Type":"ContainerStarted","Data":"f6d3f1c177c4092470f4384df5dd709fe648e4bd337ecb7f3656a80524274012"}
Oct 30 18:50:04.619988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.619971    5065 scope.go:122] "RemoveContainer" containerID="3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"
Oct 30 18:50:04.620283 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:04.620260    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e\": container with ID starting with 3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e not found: ID does not exist" containerID="3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"
Oct 30 18:50:04.620381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.620288    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e"} err="failed to get container status \"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e\": rpc error: code = NotFound desc = could not find container \"3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e\": container with ID starting with 3b16b53d51579982d432883d72eef294c9eb3981a24cfe02db0cd2c56589f33e not found: ID does not exist"
Oct 30 18:50:04.634733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.634689    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46" podStartSLOduration=2.634675365 podStartE2EDuration="2.634675365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:04.630117292 +0000 UTC m=+1024.893582273" watchObservedRunningTime="2026-10-30 18:50:04.634675365 +0000 UTC m=+1024.898140349"
Oct 30 18:50:04.713881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.713838    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:50:04.719105 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:04.719063    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58158b-5cc4bf65b4-clkpx"]
Oct 30 18:50:05.619191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:05.619157    5065 generic.go:356] "Generic (PLEG): container finished" podID="706d972e-87f6-4e69-b5b8-e085a5fd2f23" containerID="f6d3f1c177c4092470f4384df5dd709fe648e4bd337ecb7f3656a80524274012" exitCode=0
Oct 30 18:50:05.619622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:05.619238    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46" event={"ID":"706d972e-87f6-4e69-b5b8-e085a5fd2f23","Type":"ContainerDied","Data":"f6d3f1c177c4092470f4384df5dd709fe648e4bd337ecb7f3656a80524274012"}
Oct 30 18:50:05.896285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:05.896194    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2081066-aefb-4fbb-b638-86b444a07d6b" path="/var/lib/kubelet/pods/e2081066-aefb-4fbb-b638-86b444a07d6b/volumes"
Oct 30 18:50:06.903679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.897649    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:06.985610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.985574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\" (UniqueName: \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\") pod \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") "
Oct 30 18:50:06.985783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.985626    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\") pod \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") "
Oct 30 18:50:06.985855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.985793    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\") pod \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\" (UID: \"706d972e-87f6-4e69-b5b8-e085a5fd2f23\") "
Oct 30 18:50:06.989025 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.988974    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca" pod "706d972e-87f6-4e69-b5b8-e085a5fd2f23" (UID: "706d972e-87f6-4e69-b5b8-e085a5fd2f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:06.989566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.989541    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr" pod "706d972e-87f6-4e69-b5b8-e085a5fd2f23" (UID: "706d972e-87f6-4e69-b5b8-e085a5fd2f23"). InnerVolumeSpecName "kube-api-access-nf2pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:06.991248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:06.991224    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca" pod "706d972e-87f6-4e69-b5b8-e085a5fd2f23" (UID: "706d972e-87f6-4e69-b5b8-e085a5fd2f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:07.088432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.088398    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.088432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.088431    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2pr\" (UniqueName: \"kubernetes.io/projected/706d972e-87f6-4e69-b5b8-e085a5fd2f23-kube-api-access-nf2pr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.088682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.088448    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d972e-87f6-4e69-b5b8-e085a5fd2f23-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.452133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.451935    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603556    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603644    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603691    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603829    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603857    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603888    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.603917    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\") pod \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\" (UID: \"adc8d082-0ce1-48da-bf12-2d3b1470e8af\") "
Oct 30 18:50:07.605048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.604634    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:07.605865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.605050    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:07.605865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.605162    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:07.605865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.605207    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.605865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.605428    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:07.605865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.605734    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:07.607897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.607876    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "kube-api-access-zfchh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:07.607994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.607935    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:07.609081 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.609056    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:07.609315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.609291    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy" pod "adc8d082-0ce1-48da-bf12-2d3b1470e8af" (UID: "adc8d082-0ce1-48da-bf12-2d3b1470e8af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:07.645838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.645805    5065 generic.go:356] "Generic (PLEG): container finished" podID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerID="a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74" exitCode=0
Oct 30 18:50:07.646034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.645879    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"
Oct 30 18:50:07.646034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.645903    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerDied","Data":"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"}
Oct 30 18:50:07.646034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.645959    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h" event={"ID":"adc8d082-0ce1-48da-bf12-2d3b1470e8af","Type":"ContainerDied","Data":"0572bc0d42367968065b5e1e6652e5ad2aa83ddacd14520e0afd06dd64c5a7b1"}
Oct 30 18:50:07.646034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.645973    5065 scope.go:122] "RemoveContainer" containerID="a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"
Oct 30 18:50:07.647845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.647802    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46" event={"ID":"706d972e-87f6-4e69-b5b8-e085a5fd2f23","Type":"ContainerDied","Data":"c76d71d9a427f067ac53a8729b559ac52e262031e39323783f169c556b60608d"}
Oct 30 18:50:07.647845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.647844    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76d71d9a427f067ac53a8729b559ac52e262031e39323783f169c556b60608d"
Oct 30 18:50:07.648022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.647898    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"
Oct 30 18:50:07.661522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.661235    5065 scope.go:122] "RemoveContainer" containerID="650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"
Oct 30 18:50:07.675208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.675188    5065 scope.go:122] "RemoveContainer" containerID="a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"
Oct 30 18:50:07.675528 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:07.675505    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74\": container with ID starting with a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74 not found: ID does not exist" containerID="a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"
Oct 30 18:50:07.675606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.675537    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74"} err="failed to get container status \"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74\": rpc error: code = NotFound desc = could not find container \"a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74\": container with ID starting with a9d4bef9016044dd66948bc48f8ffc7c4e1980bc5461bb305ec5a82dba9c1b74 not found: ID does not exist"
Oct 30 18:50:07.675606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.675557    5065 scope.go:122] "RemoveContainer" containerID="650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"
Oct 30 18:50:07.675866 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:07.675849    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b\": container with ID starting with 650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b not found: ID does not exist" containerID="650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"
Oct 30 18:50:07.675929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.675871    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b"} err="failed to get container status \"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b\": rpc error: code = NotFound desc = could not find container \"650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b\": container with ID starting with 650cbfbbd3f0be182b80aef24a74b8ca36651dc4db41ed7a6bf6b25beb08530b not found: ID does not exist"
Oct 30 18:50:07.680957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.680930    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:50:07.692062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.692038    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4d94b-openshift-default-5f6bb547cf-7fx9h"]
Oct 30 18:50:07.707161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707136    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707161    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707171    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707180    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707189    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfchh\" (UniqueName: \"kubernetes.io/projected/adc8d082-0ce1-48da-bf12-2d3b1470e8af-kube-api-access-zfchh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707197    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707205    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.707284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.707213    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc8d082-0ce1-48da-bf12-2d3b1470e8af-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:07.906543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:07.906451    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" path="/var/lib/kubelet/pods/adc8d082-0ce1-48da-bf12-2d3b1470e8af/volumes"
Oct 30 18:50:10.683474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.683384    5065 generic.go:356] "Generic (PLEG): container finished" podID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerID="8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74" exitCode=0
Oct 30 18:50:10.683474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.683453    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerDied","Data":"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"}
Oct 30 18:50:10.684060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.683863    5065 scope.go:122] "RemoveContainer" containerID="8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"
Oct 30 18:50:10.687233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.687194    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:50:10.687360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.687287    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:50:10.687360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:10.687337    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:50:11.816110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:11.795801    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerStarted","Data":"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"}
Oct 30 18:50:11.857814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:11.857781    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"]
Oct 30 18:50:11.871561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:11.871532    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f4b265-k7vqk"]
Oct 30 18:50:11.940640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:11.940595    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b02b5fa-bbe9-4e7f-a300-ad5777189c87" path="/var/lib/kubelet/pods/6b02b5fa-bbe9-4e7f-a300-ad5777189c87/volumes"
Oct 30 18:50:12.366151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:12.363699    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:50:12.665548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:12.665466    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:50:12.665904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:12.665808    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:50:12.666119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:12.666073    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 18:50:13.664384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:13.664350    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:50:13.664776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:13.664406    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 18:50:14.664457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:14.664418    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:50:14.664858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:14.664508    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 18:50:15.671112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:15.669275    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:50:15.671112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:15.669354    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 18:50:16.349206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.349171    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"]
Oct 30 18:50:16.349953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.349934    5065 cpu_manager.go:414] "containerMap: removing container" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.349953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.349953    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.350066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.349971    5065 cpu_manager.go:414] "containerMap: removing container" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.350066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.349977    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.350066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350046    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e2081066-aefb-4fbb-b638-86b444a07d6b" containerName="payload-processing"
Oct 30 18:50:16.350066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350053    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2081066-aefb-4fbb-b638-86b444a07d6b" containerName="payload-processing"
Oct 30 18:50:16.350066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350065    5065 cpu_manager.go:414] "containerMap: removing container" podUID="706d972e-87f6-4e69-b5b8-e085a5fd2f23" containerName="revoke-keys"
Oct 30 18:50:16.350245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350069    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706d972e-87f6-4e69-b5b8-e085a5fd2f23" containerName="revoke-keys"
Oct 30 18:50:16.350245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350199    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2081066-aefb-4fbb-b638-86b444a07d6b" containerName="payload-processing"
Oct 30 18:50:16.350245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350211    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="706d972e-87f6-4e69-b5b8-e085a5fd2f23" containerName="revoke-keys"
Oct 30 18:50:16.350245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350220    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.350245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.350228    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc8d082-0ce1-48da-bf12-2d3b1470e8af" containerName="istio-proxy"
Oct 30 18:50:16.353714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.353696    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.359140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.358902    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:16.359140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.359005    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:50:16.365619 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.365581    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"]
Oct 30 18:50:16.440585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.440541    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.440810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.440789    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bpc\" (UniqueName: \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.441009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.440993    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.542047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.542014    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.542047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.542058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.542297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.542196    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bpc\" (UniqueName: \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.542850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.542786    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.544630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.544606    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.555581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.555557    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bpc\" (UniqueName: \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-zrzj9\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:16.664171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.664068    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:50:16.664171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.664150    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 18:50:16.675646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:16.675608    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:17.035785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.022112    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"]
Oct 30 18:50:17.149010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.148983    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9" event={"ID":"1246574d-8861-40e2-b24f-c213af25c26a","Type":"ContainerStarted","Data":"0a6b14736c30151932141df69ab56dd3b676579caaa58304fae8050cb7adb004"}
Oct 30 18:50:17.364543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.364495    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" podUID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" containerName="payload-pre-processing" containerID="cri-o://baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875" gracePeriod=25
Oct 30 18:50:17.671712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.671505    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:50:17.672953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.672804    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:50:17.674054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.674029    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:50:17.728483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.728453    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:50:17.867875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.867792    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\" (UniqueName: \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\") pod \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") "
Oct 30 18:50:17.868061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.868046    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\") pod \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\" (UID: \"ef8a87b0-8479-4b99-b6e5-43438ff2e45e\") "
Oct 30 18:50:17.868519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.868492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume" pod "ef8a87b0-8479-4b99-b6e5-43438ff2e45e" (UID: "ef8a87b0-8479-4b99-b6e5-43438ff2e45e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:17.868721 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.868689    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:17.870125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.870065    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq" pod "ef8a87b0-8479-4b99-b6e5-43438ff2e45e" (UID: "ef8a87b0-8479-4b99-b6e5-43438ff2e45e"). InnerVolumeSpecName "kube-api-access-xg5qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:17.974334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:17.974303    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5qq\" (UniqueName: \"kubernetes.io/projected/ef8a87b0-8479-4b99-b6e5-43438ff2e45e-kube-api-access-xg5qq\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:18.201224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.201137    5065 generic.go:356] "Generic (PLEG): container finished" podID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" containerID="baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875" exitCode=0
Oct 30 18:50:18.201610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.201432    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" event={"ID":"ef8a87b0-8479-4b99-b6e5-43438ff2e45e","Type":"ContainerDied","Data":"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"}
Oct 30 18:50:18.202250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.201453    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"
Oct 30 18:50:18.202250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.202228    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6" event={"ID":"ef8a87b0-8479-4b99-b6e5-43438ff2e45e","Type":"ContainerDied","Data":"7679062b3fdb22d009b391012704a8a5fcdbfd63e8fe8188387a7cc3ea2b568a"}
Oct 30 18:50:18.202450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.202271    5065 scope.go:122] "RemoveContainer" containerID="baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"
Oct 30 18:50:18.211246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.211197    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9" event={"ID":"1246574d-8861-40e2-b24f-c213af25c26a","Type":"ContainerStarted","Data":"71a2c0cc3ce6920487fc8bdc710f5bb43bdc88d57e3821d874f477dc91bb6bbe"}
Oct 30 18:50:18.235198 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.235157    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9" podStartSLOduration=2.235140431 podStartE2EDuration="2.235140431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:18.23067951 +0000 UTC m=+1038.494144470" watchObservedRunningTime="2026-10-30 18:50:18.235140431 +0000 UTC m=+1038.498605380"
Oct 30 18:50:18.251454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.251427    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"]
Oct 30 18:50:18.276284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.270404    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58158b-62vn7"]
Oct 30 18:50:18.291686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.291603    5065 scope.go:122] "RemoveContainer" containerID="baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"
Oct 30 18:50:18.292119 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:18.292033    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875\": container with ID starting with baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875 not found: ID does not exist" containerID="baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"
Oct 30 18:50:18.292119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.292097    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875"} err="failed to get container status \"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875\": rpc error: code = NotFound desc = could not find container \"baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875\": container with ID starting with baabbb44788b08c0d659ede606c99f57f5628397c0694d48ad0ea64e7c554875 not found: ID does not exist"
Oct 30 18:50:18.305397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.305364    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:50:18.312337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:18.312314    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f44051-5fb6574499-rtnc6"]
Oct 30 18:50:19.225413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:19.225377    5065 generic.go:356] "Generic (PLEG): container finished" podID="1246574d-8861-40e2-b24f-c213af25c26a" containerID="71a2c0cc3ce6920487fc8bdc710f5bb43bdc88d57e3821d874f477dc91bb6bbe" exitCode=0
Oct 30 18:50:19.226023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:19.225776    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9" event={"ID":"1246574d-8861-40e2-b24f-c213af25c26a","Type":"ContainerDied","Data":"71a2c0cc3ce6920487fc8bdc710f5bb43bdc88d57e3821d874f477dc91bb6bbe"}
Oct 30 18:50:19.893054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:19.893024    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98693852-1b9e-4647-8603-f9a84aafe4ca" path="/var/lib/kubelet/pods/98693852-1b9e-4647-8603-f9a84aafe4ca/volumes"
Oct 30 18:50:19.893592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:19.893572    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" path="/var/lib/kubelet/pods/ef8a87b0-8479-4b99-b6e5-43438ff2e45e/volumes"
Oct 30 18:50:20.103661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.103620    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"]
Oct 30 18:50:20.104745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.104720    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" containerName="payload-pre-processing"
Oct 30 18:50:20.104864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.104745    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" containerName="payload-pre-processing"
Oct 30 18:50:20.104984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.104970    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8a87b0-8479-4b99-b6e5-43438ff2e45e" containerName="payload-pre-processing"
Oct 30 18:50:20.111160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.111139    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.116202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.116177    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"]
Oct 30 18:50:20.236076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.236043    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.236469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.236160    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.236469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.236329    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svfg\" (UniqueName: \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.337654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.337618    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.337815 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.337670    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.337890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.337841    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svfg\" (UniqueName: \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.338636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.338607    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.340450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.340424    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.352634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.352602    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svfg\" (UniqueName: \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\") pod \"maas-api-revoke-keys-e2e-api-b-30970e-4nld7\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.422648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.422618    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:20.431544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.431521    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:20.546229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.546017    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\") pod \"1246574d-8861-40e2-b24f-c213af25c26a\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") "
Oct 30 18:50:20.547616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.547559    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca" pod "1246574d-8861-40e2-b24f-c213af25c26a" (UID: "1246574d-8861-40e2-b24f-c213af25c26a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:20.549403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.547903    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\" (UniqueName: \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\") pod \"1246574d-8861-40e2-b24f-c213af25c26a\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") "
Oct 30 18:50:20.549403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.548111    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\") pod \"1246574d-8861-40e2-b24f-c213af25c26a\" (UID: \"1246574d-8861-40e2-b24f-c213af25c26a\") "
Oct 30 18:50:20.549403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.548870    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:20.586653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.586584    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc" pod "1246574d-8861-40e2-b24f-c213af25c26a" (UID: "1246574d-8861-40e2-b24f-c213af25c26a"). InnerVolumeSpecName "kube-api-access-w6bpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:20.592972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.592934    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca" pod "1246574d-8861-40e2-b24f-c213af25c26a" (UID: "1246574d-8861-40e2-b24f-c213af25c26a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:20.651952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.651528    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bpc\" (UniqueName: \"kubernetes.io/projected/1246574d-8861-40e2-b24f-c213af25c26a-kube-api-access-w6bpc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:20.651952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.651573    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1246574d-8861-40e2-b24f-c213af25c26a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:20.738565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:20.738334    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"]
Oct 30 18:50:20.743015 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:50:20.742977    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144b9ba5_bcea_47f3_8d83_6b3f6dfe7b13.slice/crio-eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37 WatchSource:0}: Error finding container eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37: Status 404 returned error can't find the container with id eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37
Oct 30 18:50:21.244591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.244560    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7" event={"ID":"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13","Type":"ContainerStarted","Data":"4d0e3e61cb2b82f16767573f4379b2d15b45abdb40ef893f8629fade3f0f303a"}
Oct 30 18:50:21.244961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.244610    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7" event={"ID":"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13","Type":"ContainerStarted","Data":"eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37"}
Oct 30 18:50:21.246256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.246229    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"
Oct 30 18:50:21.246365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.246233    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9" event={"ID":"1246574d-8861-40e2-b24f-c213af25c26a","Type":"ContainerDied","Data":"0a6b14736c30151932141df69ab56dd3b676579caaa58304fae8050cb7adb004"}
Oct 30 18:50:21.246365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.246347    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6b14736c30151932141df69ab56dd3b676579caaa58304fae8050cb7adb004"
Oct 30 18:50:21.273692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:21.273648    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7" podStartSLOduration=1.27363645 podStartE2EDuration="1.27363645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:21.267382125 +0000 UTC m=+1041.530847088" watchObservedRunningTime="2026-10-30 18:50:21.27363645 +0000 UTC m=+1041.537101398"
Oct 30 18:50:22.256116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:22.256011    5065 generic.go:356] "Generic (PLEG): container finished" podID="144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" containerID="4d0e3e61cb2b82f16767573f4379b2d15b45abdb40ef893f8629fade3f0f303a" exitCode=0
Oct 30 18:50:22.256116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:22.256063    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7" event={"ID":"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13","Type":"ContainerDied","Data":"4d0e3e61cb2b82f16767573f4379b2d15b45abdb40ef893f8629fade3f0f303a"}
Oct 30 18:50:23.058543 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:23.058511    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:50:23.508676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.508652    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:23.611238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.611199    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\") pod \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") "
Oct 30 18:50:23.611413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.611308    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\" (UniqueName: \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\") pod \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") "
Oct 30 18:50:23.611509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.611483    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\") pod \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\" (UID: \"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13\") "
Oct 30 18:50:23.611577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.611534    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca" pod "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" (UID: "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:23.612328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.612300    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:23.613623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.613588    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca" pod "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" (UID: "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:23.613735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.613621    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg" pod "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" (UID: "144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13"). InnerVolumeSpecName "kube-api-access-7svfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:23.713385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.713323    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:23.713385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:23.713353    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svfg\" (UniqueName: \"kubernetes.io/projected/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13-kube-api-access-7svfg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.342913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:24.342888    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"
Oct 30 18:50:24.343126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:24.342927    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7" event={"ID":"144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13","Type":"ContainerDied","Data":"eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37"}
Oct 30 18:50:24.343126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:24.343014    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5065332aa7c57846d4006783ad72e36717b3317f7acf69553400748fd50f37"
Oct 30 18:50:25.033145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:25.033113    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"]
Oct 30 18:50:25.038362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:25.038337    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4255e13f-p87tk"]
Oct 30 18:50:25.877953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:25.877922    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:50:25.878215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:25.878190    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy" containerID="cri-o://e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0" gracePeriod=30
Oct 30 18:50:25.894336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:25.894309    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64184c22-bd6b-41c4-bba6-38c89c3ba6fd" path="/var/lib/kubelet/pods/64184c22-bd6b-41c4-bba6-38c89c3ba6fd/volumes"
Oct 30 18:50:26.497682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.497650    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"]
Oct 30 18:50:26.498385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498369    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1246574d-8861-40e2-b24f-c213af25c26a" containerName="revoke-keys"
Oct 30 18:50:26.498429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498386    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1246574d-8861-40e2-b24f-c213af25c26a" containerName="revoke-keys"
Oct 30 18:50:26.498429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498411    5065 cpu_manager.go:414] "containerMap: removing container" podUID="144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" containerName="revoke-keys"
Oct 30 18:50:26.498429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498416    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" containerName="revoke-keys"
Oct 30 18:50:26.498539 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498530    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1246574d-8861-40e2-b24f-c213af25c26a" containerName="revoke-keys"
Oct 30 18:50:26.498577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.498541    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" containerName="revoke-keys"
Oct 30 18:50:26.502790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.502771    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.512456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.512433    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"]
Oct 30 18:50:26.520020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.519996    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:26.520324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.520299    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:50:26.550452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.550429    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"]
Oct 30 18:50:26.555288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.555268    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.569818 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.569796    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"]
Oct 30 18:50:26.618716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.618671    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.619069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.619036    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.619401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.619373    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwtl\" (UniqueName: \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.720479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720450    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.720674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720648    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.720749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720734    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4q9\" (UniqueName: \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.720794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720765    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.720921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720897    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.720997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.720979    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwtl\" (UniqueName: \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.721397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.721371    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.723871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.723849    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.735718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.735688    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwtl\" (UniqueName: \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\") pod \"maas-api-revoke-keys-e2e-api-b-f9521e-ths4c\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.822485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.822457    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.822622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.822599    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4q9\" (UniqueName: \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.822669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.822657    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.823322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.823298    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.824895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.824877    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.825048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.825036    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:26.835927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.835905    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4q9\" (UniqueName: \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\") pod \"maas-api-revoke-sub-222a4ad8-p8phl\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:26.883108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:26.880781    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:27.399211 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:50:27.399178    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69882c4_a2d2_4425_bbba_22a727d16068.slice/crio-7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787 WatchSource:0}: Error finding container 7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787: Status 404 returned error can't find the container with id 7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787
Oct 30 18:50:27.399933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:27.399908    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"]
Oct 30 18:50:27.435475 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:50:27.435446    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175fb5b0_78d1_4d37_8940_795baba6a1b3.slice/crio-459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b WatchSource:0}: Error finding container 459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b: Status 404 returned error can't find the container with id 459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b
Oct 30 18:50:27.439122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:27.438494    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"]
Oct 30 18:50:28.151439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.151352    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"]
Oct 30 18:50:28.160136 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.160079    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f44051-rpp46"]
Oct 30 18:50:28.410179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.407046    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl" event={"ID":"175fb5b0-78d1-4d37-8940-795baba6a1b3","Type":"ContainerStarted","Data":"a1e34f55a48a094c447239d1d2bf70a87688bd40c9ef45bb63b496f826fa2393"}
Oct 30 18:50:28.410179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.407146    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl" event={"ID":"175fb5b0-78d1-4d37-8940-795baba6a1b3","Type":"ContainerStarted","Data":"459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b"}
Oct 30 18:50:28.410921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.410892    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c" event={"ID":"e69882c4-a2d2-4425-bbba-22a727d16068","Type":"ContainerStarted","Data":"20ee1a06ddb574c22a78370d10614133665f43cafc25b91ba6b4eefe7c6c671f"}
Oct 30 18:50:28.410982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.410948    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c" event={"ID":"e69882c4-a2d2-4425-bbba-22a727d16068","Type":"ContainerStarted","Data":"7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787"}
Oct 30 18:50:28.428927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.428882    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl" podStartSLOduration=2.428871022 podStartE2EDuration="2.428871022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.428018752 +0000 UTC m=+1048.691483702" watchObservedRunningTime="2026-10-30 18:50:28.428871022 +0000 UTC m=+1048.692336008"
Oct 30 18:50:28.457017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:28.456971    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c" podStartSLOduration=2.456955414 podStartE2EDuration="2.456955414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.449046462 +0000 UTC m=+1048.712511425" watchObservedRunningTime="2026-10-30 18:50:28.456955414 +0000 UTC m=+1048.720420366"
Oct 30 18:50:29.425825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:29.425786    5065 generic.go:356] "Generic (PLEG): container finished" podID="175fb5b0-78d1-4d37-8940-795baba6a1b3" containerID="a1e34f55a48a094c447239d1d2bf70a87688bd40c9ef45bb63b496f826fa2393" exitCode=0
Oct 30 18:50:29.426322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:29.425854    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl" event={"ID":"175fb5b0-78d1-4d37-8940-795baba6a1b3","Type":"ContainerDied","Data":"a1e34f55a48a094c447239d1d2bf70a87688bd40c9ef45bb63b496f826fa2393"}
Oct 30 18:50:29.428126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:29.428062    5065 generic.go:356] "Generic (PLEG): container finished" podID="e69882c4-a2d2-4425-bbba-22a727d16068" containerID="20ee1a06ddb574c22a78370d10614133665f43cafc25b91ba6b4eefe7c6c671f" exitCode=0
Oct 30 18:50:29.428262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:29.428160    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c" event={"ID":"e69882c4-a2d2-4425-bbba-22a727d16068","Type":"ContainerDied","Data":"20ee1a06ddb574c22a78370d10614133665f43cafc25b91ba6b4eefe7c6c671f"}
Oct 30 18:50:29.892635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:29.892607    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706d972e-87f6-4e69-b5b8-e085a5fd2f23" path="/var/lib/kubelet/pods/706d972e-87f6-4e69-b5b8-e085a5fd2f23/volumes"
Oct 30 18:50:30.281882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.281849    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:50:30.313383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.313353    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:50:30.861983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.861952    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:30.935715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.935685    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\") pod \"e69882c4-a2d2-4425-bbba-22a727d16068\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") "
Oct 30 18:50:30.935840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.935810    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\" (UniqueName: \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\") pod \"e69882c4-a2d2-4425-bbba-22a727d16068\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") "
Oct 30 18:50:30.935966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.935949    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\") pod \"e69882c4-a2d2-4425-bbba-22a727d16068\" (UID: \"e69882c4-a2d2-4425-bbba-22a727d16068\") "
Oct 30 18:50:30.937346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.937211    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca" pod "e69882c4-a2d2-4425-bbba-22a727d16068" (UID: "e69882c4-a2d2-4425-bbba-22a727d16068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:30.938930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.938876    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca" pod "e69882c4-a2d2-4425-bbba-22a727d16068" (UID: "e69882c4-a2d2-4425-bbba-22a727d16068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:30.940076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.940052    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl" pod "e69882c4-a2d2-4425-bbba-22a727d16068" (UID: "e69882c4-a2d2-4425-bbba-22a727d16068"). InnerVolumeSpecName "kube-api-access-6cwtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:30.992228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:30.992199    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:31.043840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.043491    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\") pod \"175fb5b0-78d1-4d37-8940-795baba6a1b3\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") "
Oct 30 18:50:31.043840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.043575    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\") pod \"175fb5b0-78d1-4d37-8940-795baba6a1b3\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") "
Oct 30 18:50:31.043840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.043615    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\" (UniqueName: \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\") pod \"175fb5b0-78d1-4d37-8940-795baba6a1b3\" (UID: \"175fb5b0-78d1-4d37-8940-795baba6a1b3\") "
Oct 30 18:50:31.044263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.043907    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"]
Oct 30 18:50:31.044363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.044336    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.044447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.044364    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwtl\" (UniqueName: \"kubernetes.io/projected/e69882c4-a2d2-4425-bbba-22a727d16068-kube-api-access-6cwtl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.044447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.044380    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69882c4-a2d2-4425-bbba-22a727d16068-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.045269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.045188    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca" pod "175fb5b0-78d1-4d37-8940-795baba6a1b3" (UID: "175fb5b0-78d1-4d37-8940-795baba6a1b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:31.047315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.047292    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca" pod "175fb5b0-78d1-4d37-8940-795baba6a1b3" (UID: "175fb5b0-78d1-4d37-8940-795baba6a1b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:31.047454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.047353    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9" pod "175fb5b0-78d1-4d37-8940-795baba6a1b3" (UID: "175fb5b0-78d1-4d37-8940-795baba6a1b3"). InnerVolumeSpecName "kube-api-access-gw4q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:31.058065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.058039    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91fc570-xlrrq"]
Oct 30 18:50:31.147038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.147002    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.147038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.147038    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175fb5b0-78d1-4d37-8940-795baba6a1b3-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.147307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.147057    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4q9\" (UniqueName: \"kubernetes.io/projected/175fb5b0-78d1-4d37-8940-795baba6a1b3-kube-api-access-gw4q9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.189783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.189757    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:50:31.247882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.247851    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248244    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248289    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248335    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248424    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248494    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248534    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248564    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248571    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:31.248637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248593    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\") pod \"45e77fa7-04d1-48c8-995d-f8137d457d22\" (UID: \"45e77fa7-04d1-48c8-995d-f8137d457d22\") "
Oct 30 18:50:31.248956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248775    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:31.249014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.248955    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:31.250331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.249365    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.250331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.249387    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.250331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.249402    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.250331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.249604    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:31.250331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.249886    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:31.251948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.251910    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "kube-api-access-fm4cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:31.252078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.252025    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:31.252914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.252886    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:31.253751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.253728    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo" pod "45e77fa7-04d1-48c8-995d-f8137d457d22" (UID: "45e77fa7-04d1-48c8-995d-f8137d457d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:31.351586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351463    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.351586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351541    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.351586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351558    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.351586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351572    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e77fa7-04d1-48c8-995d-f8137d457d22-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.351845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351586    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e77fa7-04d1-48c8-995d-f8137d457d22-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.351845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.351607    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4cz\" (UniqueName: \"kubernetes.io/projected/45e77fa7-04d1-48c8-995d-f8137d457d22-kube-api-access-fm4cz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:31.449552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.449527    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"
Oct 30 18:50:31.449745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.449560    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl" event={"ID":"175fb5b0-78d1-4d37-8940-795baba6a1b3","Type":"ContainerDied","Data":"459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b"}
Oct 30 18:50:31.449745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.449626    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459bc346d03f10ed196a287b7262fe4564f3a5a22f24e89018fefea5dd0de27b"
Oct 30 18:50:31.451289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.451268    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"
Oct 30 18:50:31.451417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.451289    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c" event={"ID":"e69882c4-a2d2-4425-bbba-22a727d16068","Type":"ContainerDied","Data":"7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787"}
Oct 30 18:50:31.451417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.451341    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c774bfc1472455b52abf93b3ea5e6bd167064c48c02d36dca4f3168b81f9787"
Oct 30 18:50:31.453571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.453549    5065 generic.go:356] "Generic (PLEG): container finished" podID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerID="e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0" exitCode=0
Oct 30 18:50:31.453675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.453608    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"
Oct 30 18:50:31.453675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.453632    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerDied","Data":"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"}
Oct 30 18:50:31.453763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.453693    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l" event={"ID":"45e77fa7-04d1-48c8-995d-f8137d457d22","Type":"ContainerDied","Data":"72c7cf897bddc69d4c7c80862e140290c1fe7986134ba3a334043b71a4925d52"}
Oct 30 18:50:31.453763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.453726    5065 scope.go:122] "RemoveContainer" containerID="e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"
Oct 30 18:50:31.472222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.472201    5065 scope.go:122] "RemoveContainer" containerID="17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"
Oct 30 18:50:31.484145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.484080    5065 scope.go:122] "RemoveContainer" containerID="e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"
Oct 30 18:50:31.484488 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:31.484468    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0\": container with ID starting with e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0 not found: ID does not exist" containerID="e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"
Oct 30 18:50:31.484565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.484498    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0"} err="failed to get container status \"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0\": rpc error: code = NotFound desc = could not find container \"e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0\": container with ID starting with e992c5902c7dbe8699107cc9ed7a7a5ad32cb6e037fdabf70219121ca164a3d0 not found: ID does not exist"
Oct 30 18:50:31.484565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.484515    5065 scope.go:122] "RemoveContainer" containerID="17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"
Oct 30 18:50:31.484837 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:31.484804    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444\": container with ID starting with 17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444 not found: ID does not exist" containerID="17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"
Oct 30 18:50:31.484928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.484838    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444"} err="failed to get container status \"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444\": rpc error: code = NotFound desc = could not find container \"17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444\": container with ID starting with 17e691b4636c1cec4fef8af0ed1732a7b9a5f66d5109e5ed6e1d26d3020ff444 not found: ID does not exist"
Oct 30 18:50:31.505508 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.505483    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:50:31.514780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.514760    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-58158b-openshift-default-6fdc6b584c-s9s5l"]
Oct 30 18:50:31.893211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.893170    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" path="/var/lib/kubelet/pods/45e77fa7-04d1-48c8-995d-f8137d457d22/volumes"
Oct 30 18:50:31.893686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:31.893664    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f337a44-f9c4-4377-9480-910fd1999dbe" path="/var/lib/kubelet/pods/9f337a44-f9c4-4377-9480-910fd1999dbe/volumes"
Oct 30 18:50:35.283035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.282989    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" podUID="bd665943-9d99-46ac-b5f4-15c785bb39f2" containerName="payload-processing" containerID="cri-o://52428685764b874f4907873975228bec59af11b417ad4e816fe5fc1a02cb81ea" gracePeriod=25
Oct 30 18:50:35.314792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.314757    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" podUID="23357381-db62-4c25-a29a-8305dfca964b" containerName="payload-pre-processing" containerID="cri-o://38b4593cc811079b005f2f7b084a69e9dde1219aea7a60d1987aee9173be24d1" gracePeriod=25
Oct 30 18:50:35.503141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.496611    5065 generic.go:356] "Generic (PLEG): container finished" podID="23357381-db62-4c25-a29a-8305dfca964b" containerID="38b4593cc811079b005f2f7b084a69e9dde1219aea7a60d1987aee9173be24d1" exitCode=0
Oct 30 18:50:35.503141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.496693    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" event={"ID":"23357381-db62-4c25-a29a-8305dfca964b","Type":"ContainerDied","Data":"38b4593cc811079b005f2f7b084a69e9dde1219aea7a60d1987aee9173be24d1"}
Oct 30 18:50:35.503141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.499288    5065 generic.go:356] "Generic (PLEG): container finished" podID="bd665943-9d99-46ac-b5f4-15c785bb39f2" containerID="52428685764b874f4907873975228bec59af11b417ad4e816fe5fc1a02cb81ea" exitCode=0
Oct 30 18:50:35.503141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.499384    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" event={"ID":"bd665943-9d99-46ac-b5f4-15c785bb39f2","Type":"ContainerDied","Data":"52428685764b874f4907873975228bec59af11b417ad4e816fe5fc1a02cb81ea"}
Oct 30 18:50:35.644625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.644594    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:50:35.657900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.657879    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:50:35.722638 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.722599    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\") pod \"23357381-db62-4c25-a29a-8305dfca964b\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") "
Oct 30 18:50:35.722824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.722666    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\" (UniqueName: \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\") pod \"23357381-db62-4c25-a29a-8305dfca964b\" (UID: \"23357381-db62-4c25-a29a-8305dfca964b\") "
Oct 30 18:50:35.722824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.722721    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\") pod \"bd665943-9d99-46ac-b5f4-15c785bb39f2\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") "
Oct 30 18:50:35.722824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.722770    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\" (UniqueName: \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\") pod \"bd665943-9d99-46ac-b5f4-15c785bb39f2\" (UID: \"bd665943-9d99-46ac-b5f4-15c785bb39f2\") "
Oct 30 18:50:35.723074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.723045    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume" pod "23357381-db62-4c25-a29a-8305dfca964b" (UID: "23357381-db62-4c25-a29a-8305dfca964b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:35.723220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.723191    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume" pod "bd665943-9d99-46ac-b5f4-15c785bb39f2" (UID: "bd665943-9d99-46ac-b5f4-15c785bb39f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:35.723472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.723448    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23357381-db62-4c25-a29a-8305dfca964b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:35.723576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.723476    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd665943-9d99-46ac-b5f4-15c785bb39f2-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:35.725058 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.725035    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc" pod "23357381-db62-4c25-a29a-8305dfca964b" (UID: "23357381-db62-4c25-a29a-8305dfca964b"). InnerVolumeSpecName "kube-api-access-b48gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:35.725200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.725183    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6" pod "bd665943-9d99-46ac-b5f4-15c785bb39f2" (UID: "bd665943-9d99-46ac-b5f4-15c785bb39f2"). InnerVolumeSpecName "kube-api-access-ssjq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:35.825359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.825270    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48gc\" (UniqueName: \"kubernetes.io/projected/23357381-db62-4c25-a29a-8305dfca964b-kube-api-access-b48gc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:35.825359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:35.825300    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjq6\" (UniqueName: \"kubernetes.io/projected/bd665943-9d99-46ac-b5f4-15c785bb39f2-kube-api-access-ssjq6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:36.330437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.330407    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:50:36.331158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.331124    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" containerID="cri-o://7f08430e5f7107f047028f44860e030feb534544619f235a85c045765240ccfa" gracePeriod=30
Oct 30 18:50:36.350548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.350524    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:50:36.350709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.350692    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" containerID="cri-o://d90866b060c0c9b6fedba5997e701f46b433c329cd4c9abc82cd0434fc28c7aa" gracePeriod=30
Oct 30 18:50:36.554161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.549193    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:50:36.554161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.549279    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:36.585844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.585750    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k" event={"ID":"bd665943-9d99-46ac-b5f4-15c785bb39f2","Type":"ContainerDied","Data":"ce54ed77a93407078181b92718fad3e91c38aaba6b052ced0b0a75a67ceefd8b"}
Oct 30 18:50:36.586139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.586120    5065 scope.go:122] "RemoveContainer" containerID="52428685764b874f4907873975228bec59af11b417ad4e816fe5fc1a02cb81ea"
Oct 30 18:50:36.586429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.586414    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"
Oct 30 18:50:36.594706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.594682    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt" event={"ID":"23357381-db62-4c25-a29a-8305dfca964b","Type":"ContainerDied","Data":"7011bfbd1c708a4d3a175cf8a9cc00c9f81313d3c0b41304211a54b7ce79f175"}
Oct 30 18:50:36.595056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.595039    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"
Oct 30 18:50:36.612620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.612055    5065 scope.go:122] "RemoveContainer" containerID="38b4593cc811079b005f2f7b084a69e9dde1219aea7a60d1987aee9173be24d1"
Oct 30 18:50:36.629709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.629682    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:50:36.637430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.637402    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-30970e-7fb6f9466d-fgl2k"]
Oct 30 18:50:36.663735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.663704    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:50:36.674113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.673803    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30970e-768848cd8c-bpwlt"]
Oct 30 18:50:36.751863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:36.751828    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:50:37.339426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:37.339368    5065 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:50:37.339917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:37.339441    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:50:37.894631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:37.894599    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23357381-db62-4c25-a29a-8305dfca964b" path="/var/lib/kubelet/pods/23357381-db62-4c25-a29a-8305dfca964b/volumes"
Oct 30 18:50:37.895252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:37.895222    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd665943-9d99-46ac-b5f4-15c785bb39f2" path="/var/lib/kubelet/pods/bd665943-9d99-46ac-b5f4-15c785bb39f2/volumes"
Oct 30 18:50:41.031712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.031679    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"]
Oct 30 18:50:41.036708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.036687    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4dc494-jbkvn"]
Oct 30 18:50:41.721222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.721193    5065 generic.go:356] "Generic (PLEG): container finished" podID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerID="d90866b060c0c9b6fedba5997e701f46b433c329cd4c9abc82cd0434fc28c7aa" exitCode=0
Oct 30 18:50:41.721602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.721582    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" event={"ID":"5266cc25-e43e-4ad1-b80f-2be71aec35d7","Type":"ContainerDied","Data":"d90866b060c0c9b6fedba5997e701f46b433c329cd4c9abc82cd0434fc28c7aa"}
Oct 30 18:50:41.725117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.724791    5065 generic.go:356] "Generic (PLEG): container finished" podID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerID="7f08430e5f7107f047028f44860e030feb534544619f235a85c045765240ccfa" exitCode=0
Oct 30 18:50:41.725117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.724949    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerDied","Data":"7f08430e5f7107f047028f44860e030feb534544619f235a85c045765240ccfa"}
Oct 30 18:50:41.725117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.724994    5065 scope.go:122] "RemoveContainer" containerID="83f122f10a6f7080f06b49c72e1a0603861246cb2bcc560a259c721660f45b73"
Oct 30 18:50:41.752435 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.752133    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" podUID="d8719b6b-f2b5-433c-a68a-f310fef9838a" containerName="payload-pre-processing" containerID="cri-o://7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b" gracePeriod=25
Oct 30 18:50:41.838197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.837984    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:50:41.871064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.871043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:50:41.897932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.897179    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2760b4-cdc3-42b7-97d7-56ef8c51c337" path="/var/lib/kubelet/pods/ac2760b4-cdc3-42b7-97d7-56ef8c51c337/volumes"
Oct 30 18:50:41.953264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953179    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953215    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953233    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953251    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953283    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953328    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953346    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953364    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953411    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953438    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953474    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953549    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.953603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.954052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953666    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.954052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953771    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.954052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953815    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\") pod \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\" (UID: \"5266cc25-e43e-4ad1-b80f-2be71aec35d7\") "
Oct 30 18:50:41.954052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953879    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.954052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.953923    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\") pod \"458092dc-c9a1-4747-a2ec-56fa503b1c49\" (UID: \"458092dc-c9a1-4747-a2ec-56fa503b1c49\") "
Oct 30 18:50:41.954475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954452    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954705    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954865    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954789    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954796    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954806    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.954963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.954920    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.955696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.955544    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:41.956462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.956402    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:41.957654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.957609    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.960196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.960160    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:41.960565 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.960516    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "kube-api-access-4rbdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.960799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.960743    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.960799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.960774    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "kube-api-access-z8hgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.961055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.961031    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.967295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.967222    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.967295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.967261    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo" pod "458092dc-c9a1-4747-a2ec-56fa503b1c49" (UID: "458092dc-c9a1-4747-a2ec-56fa503b1c49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:41.968217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:41.968193    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data" pod "5266cc25-e43e-4ad1-b80f-2be71aec35d7" (UID: "5266cc25-e43e-4ad1-b80f-2be71aec35d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:42.055392 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055365    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458092dc-c9a1-4747-a2ec-56fa503b1c49-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055392 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055390    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8hgj\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-kube-api-access-z8hgj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055404    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055418    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055426    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055434    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055442    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055450    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055460    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055469    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055479    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055491    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055500    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055508    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458092dc-c9a1-4747-a2ec-56fa503b1c49-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055518    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266cc25-e43e-4ad1-b80f-2be71aec35d7-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055526    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266cc25-e43e-4ad1-b80f-2be71aec35d7-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055535    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbdv\" (UniqueName: \"kubernetes.io/projected/5266cc25-e43e-4ad1-b80f-2be71aec35d7-kube-api-access-4rbdv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.055782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.055542    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458092dc-c9a1-4747-a2ec-56fa503b1c49-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.083338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.083310    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:50:42.156485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.156451    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\") pod \"d8719b6b-f2b5-433c-a68a-f310fef9838a\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") "
Oct 30 18:50:42.156712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.156695    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\" (UniqueName: \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\") pod \"d8719b6b-f2b5-433c-a68a-f310fef9838a\" (UID: \"d8719b6b-f2b5-433c-a68a-f310fef9838a\") "
Oct 30 18:50:42.156832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.156743    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume" pod "d8719b6b-f2b5-433c-a68a-f310fef9838a" (UID: "d8719b6b-f2b5-433c-a68a-f310fef9838a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:42.157333 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.157313    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8719b6b-f2b5-433c-a68a-f310fef9838a-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.159131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.159071    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb" pod "d8719b6b-f2b5-433c-a68a-f310fef9838a" (UID: "d8719b6b-f2b5-433c-a68a-f310fef9838a"). InnerVolumeSpecName "kube-api-access-b98kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:42.258716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.258690    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98kb\" (UniqueName: \"kubernetes.io/projected/d8719b6b-f2b5-433c-a68a-f310fef9838a-kube-api-access-b98kb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.737736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.737648    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"
Oct 30 18:50:42.737736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.737688    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm" event={"ID":"5266cc25-e43e-4ad1-b80f-2be71aec35d7","Type":"ContainerDied","Data":"3bd8665bd8c57370eb471883792001dbc8d3843405267a2c6823fee365b6538b"}
Oct 30 18:50:42.737960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.737755    5065 scope.go:122] "RemoveContainer" containerID="d90866b060c0c9b6fedba5997e701f46b433c329cd4c9abc82cd0434fc28c7aa"
Oct 30 18:50:42.739734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.739432    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"
Oct 30 18:50:42.739734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.739556    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4" event={"ID":"458092dc-c9a1-4747-a2ec-56fa503b1c49","Type":"ContainerDied","Data":"25b1a7f614de3bf6cfce7c5699987da37f08167054f8f794eef11cd8d38e99f9"}
Oct 30 18:50:42.741447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.741407    5065 generic.go:356] "Generic (PLEG): container finished" podID="d8719b6b-f2b5-433c-a68a-f310fef9838a" containerID="7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b" exitCode=0
Oct 30 18:50:42.741629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.741461    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"
Oct 30 18:50:42.741629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.741527    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" event={"ID":"d8719b6b-f2b5-433c-a68a-f310fef9838a","Type":"ContainerDied","Data":"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"}
Oct 30 18:50:42.741629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.741564    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb" event={"ID":"d8719b6b-f2b5-433c-a68a-f310fef9838a","Type":"ContainerDied","Data":"a89d594cb6d3097309d0f12f66849d9efa7c15e69b3220afbc7f3a72c804f20a"}
Oct 30 18:50:42.753871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.753849    5065 scope.go:122] "RemoveContainer" containerID="7f08430e5f7107f047028f44860e030feb534544619f235a85c045765240ccfa"
Oct 30 18:50:42.766171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.766153    5065 scope.go:122] "RemoveContainer" containerID="7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"
Oct 30 18:50:42.776179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.776161    5065 scope.go:122] "RemoveContainer" containerID="7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"
Oct 30 18:50:42.776480 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:42.776461    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b\": container with ID starting with 7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b not found: ID does not exist" containerID="7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"
Oct 30 18:50:42.776524 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.776486    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b"} err="failed to get container status \"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b\": rpc error: code = NotFound desc = could not find container \"7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b\": container with ID starting with 7f500af411cfc5cfe0841da0dde82b63fbed4dba531d3301a90aeb7156fe313b not found: ID does not exist"
Oct 30 18:50:42.820333 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.820307    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:50:42.825165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.825141    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-7mmxm"]
Oct 30 18:50:42.841933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.841909    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:50:42.854927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.854893    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-klsx4"]
Oct 30 18:50:42.874811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.874790    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:50:42.878902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:42.878884    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9521e-b45b65795-f57gb"]
Oct 30 18:50:43.899778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:43.899717    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" path="/var/lib/kubelet/pods/458092dc-c9a1-4747-a2ec-56fa503b1c49/volumes"
Oct 30 18:50:43.900736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:43.900711    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" path="/var/lib/kubelet/pods/5266cc25-e43e-4ad1-b80f-2be71aec35d7/volumes"
Oct 30 18:50:43.901400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:43.901376    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8719b6b-f2b5-433c-a68a-f310fef9838a" path="/var/lib/kubelet/pods/d8719b6b-f2b5-433c-a68a-f310fef9838a/volumes"
Oct 30 18:50:45.165813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.165783    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"]
Oct 30 18:50:45.170391 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.170366    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-30970e-4nld7"]
Oct 30 18:50:45.618905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.618742    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"]
Oct 30 18:50:45.626559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.626530    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-zrzj9"]
Oct 30 18:50:45.893946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.893856    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1246574d-8861-40e2-b24f-c213af25c26a" path="/var/lib/kubelet/pods/1246574d-8861-40e2-b24f-c213af25c26a/volumes"
Oct 30 18:50:45.894622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:45.894601    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13" path="/var/lib/kubelet/pods/144b9ba5-bcea-47f3-8d83-6b3f6dfe7b13/volumes"
Oct 30 18:50:50.554789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:50.553161    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"]
Oct 30 18:50:50.559989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:50.559961    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9521e-ths4c"]
Oct 30 18:50:51.892118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:51.892061    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69882c4-a2d2-4425-bbba-22a727d16068" path="/var/lib/kubelet/pods/e69882c4-a2d2-4425-bbba-22a727d16068/volumes"
Oct 30 18:50:53.066049 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:50:53.066013    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:50:56.620829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.620795    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"]
Oct 30 18:50:56.621810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621790    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e69882c4-a2d2-4425-bbba-22a727d16068" containerName="revoke-keys"
Oct 30 18:50:56.621858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621814    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69882c4-a2d2-4425-bbba-22a727d16068" containerName="revoke-keys"
Oct 30 18:50:56.621858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621841    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bd665943-9d99-46ac-b5f4-15c785bb39f2" containerName="payload-processing"
Oct 30 18:50:56.621858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621849    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd665943-9d99-46ac-b5f4-15c785bb39f2" containerName="payload-processing"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621869    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621877    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621892    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d8719b6b-f2b5-433c-a68a-f310fef9838a" containerName="payload-pre-processing"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621899    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8719b6b-f2b5-433c-a68a-f310fef9838a" containerName="payload-pre-processing"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621916    5065 cpu_manager.go:414] "containerMap: removing container" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621923    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621939    5065 cpu_manager.go:414] "containerMap: removing container" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621946    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:50:56.621958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621958    5065 cpu_manager.go:414] "containerMap: removing container" podUID="175fb5b0-78d1-4d37-8940-795baba6a1b3" containerName="revoke-keys"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621964    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175fb5b0-78d1-4d37-8940-795baba6a1b3" containerName="revoke-keys"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621978    5065 cpu_manager.go:414] "containerMap: removing container" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.621984    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622010    5065 cpu_manager.go:414] "containerMap: removing container" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622017    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622045    5065 cpu_manager.go:414] "containerMap: removing container" podUID="23357381-db62-4c25-a29a-8305dfca964b" containerName="payload-pre-processing"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622052    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23357381-db62-4c25-a29a-8305dfca964b" containerName="payload-pre-processing"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622224    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622239    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="23357381-db62-4c25-a29a-8305dfca964b" containerName="payload-pre-processing"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622250    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69882c4-a2d2-4425-bbba-22a727d16068" containerName="revoke-keys"
Oct 30 18:50:56.622260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622263    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd665943-9d99-46ac-b5f4-15c785bb39f2" containerName="payload-processing"
Oct 30 18:50:56.622570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622273    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5266cc25-e43e-4ad1-b80f-2be71aec35d7" containerName="istio-proxy"
Oct 30 18:50:56.622570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622285    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8719b6b-f2b5-433c-a68a-f310fef9838a" containerName="payload-pre-processing"
Oct 30 18:50:56.622570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622296    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.622570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622307    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="175fb5b0-78d1-4d37-8940-795baba6a1b3" containerName="revoke-keys"
Oct 30 18:50:56.622570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.622318    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="458092dc-c9a1-4747-a2ec-56fa503b1c49" containerName="istio-proxy"
Oct 30 18:50:56.629022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.628488    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.636482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.636456    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"]
Oct 30 18:50:56.637457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.637430    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:50:56.638232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.638189    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:56.705503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.705474    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgsz\" (UniqueName: \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.705653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.705523    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.705772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.705757    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.806795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.806768    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.806971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.806935    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgsz\" (UniqueName: \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.807037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.806974    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.807587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.807560    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.810098 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.810061    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.821514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.821489    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgsz\" (UniqueName: \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\") pod \"maas-api-revoke-sub-59bd1c3b-xt2ft\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:56.949272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:56.949197    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:50:57.271705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:57.271662    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"]
Oct 30 18:50:57.274649 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:50:57.274608    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dd5c56_1026_4303_8825_6912c5dbce26.slice/crio-43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144 WatchSource:0}: Error finding container 43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144: Status 404 returned error can't find the container with id 43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144
Oct 30 18:50:57.899167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:57.899032    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft" event={"ID":"d5dd5c56-1026-4303-8825-6912c5dbce26","Type":"ContainerStarted","Data":"b997927051367bb3dfa12e17002681f2297cb923c1baf3c3933666c94c06d0a1"}
Oct 30 18:50:57.899167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:57.899078    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft" event={"ID":"d5dd5c56-1026-4303-8825-6912c5dbce26","Type":"ContainerStarted","Data":"43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144"}
Oct 30 18:50:57.926304 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:57.926244    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft" podStartSLOduration=1.9262282750000002 podStartE2EDuration="1.926228275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:57.925013207 +0000 UTC m=+1078.188478155" watchObservedRunningTime="2026-10-30 18:50:57.926228275 +0000 UTC m=+1078.189693231"
Oct 30 18:50:58.916662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:58.916624    5065 generic.go:356] "Generic (PLEG): container finished" podID="d5dd5c56-1026-4303-8825-6912c5dbce26" containerID="b997927051367bb3dfa12e17002681f2297cb923c1baf3c3933666c94c06d0a1" exitCode=0
Oct 30 18:50:58.917130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:50:58.916729    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft" event={"ID":"d5dd5c56-1026-4303-8825-6912c5dbce26","Type":"ContainerDied","Data":"b997927051367bb3dfa12e17002681f2297cb923c1baf3c3933666c94c06d0a1"}
Oct 30 18:51:00.137278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.137249    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:51:00.264165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.252423    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\") pod \"d5dd5c56-1026-4303-8825-6912c5dbce26\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") "
Oct 30 18:51:00.268111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.264466    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\" (UniqueName: \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\") pod \"d5dd5c56-1026-4303-8825-6912c5dbce26\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") "
Oct 30 18:51:00.339119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.268583    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\") pod \"d5dd5c56-1026-4303-8825-6912c5dbce26\" (UID: \"d5dd5c56-1026-4303-8825-6912c5dbce26\") "
Oct 30 18:51:00.339119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.254254    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca" pod "d5dd5c56-1026-4303-8825-6912c5dbce26" (UID: "d5dd5c56-1026-4303-8825-6912c5dbce26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:00.386709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.383195    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:00.395112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.387515    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca" pod "d5dd5c56-1026-4303-8825-6912c5dbce26" (UID: "d5dd5c56-1026-4303-8825-6912c5dbce26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:00.399326 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.398475    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz" pod "d5dd5c56-1026-4303-8825-6912c5dbce26" (UID: "d5dd5c56-1026-4303-8825-6912c5dbce26"). InnerVolumeSpecName "kube-api-access-dtgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:00.485283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.485248    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5dd5c56-1026-4303-8825-6912c5dbce26-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:00.485283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.485284    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgsz\" (UniqueName: \"kubernetes.io/projected/d5dd5c56-1026-4303-8825-6912c5dbce26-kube-api-access-dtgsz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:00.951800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.951762    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft" event={"ID":"d5dd5c56-1026-4303-8825-6912c5dbce26","Type":"ContainerDied","Data":"43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144"}
Oct 30 18:51:00.952045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.952027    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ae03efaf21c451de8065b0d057f4d7d5ef0c22cf319af1de0354f47b55d144"
Oct 30 18:51:00.952243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:00.952221    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"
Oct 30 18:51:01.247852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.247754    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:51:01.248406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.248309    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" containerID="cri-o://9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e" gracePeriod=30
Oct 30 18:51:01.476140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.476075    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:51:01.477250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.477222    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d5dd5c56-1026-4303-8825-6912c5dbce26" containerName="revoke-keys"
Oct 30 18:51:01.477250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.477247    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dd5c56-1026-4303-8825-6912c5dbce26" containerName="revoke-keys"
Oct 30 18:51:01.477468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.477429    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e77fa7-04d1-48c8-995d-f8137d457d22" containerName="istio-proxy"
Oct 30 18:51:01.477468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.477454    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dd5c56-1026-4303-8825-6912c5dbce26" containerName="revoke-keys"
Oct 30 18:51:01.483145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.483123    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.493571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.493547    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0d540b-openshift-default-dockercfg-5627z\""
Oct 30 18:51:01.514713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.514681    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.638688    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5bp\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.638756    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.638808    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.638905    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.638946    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.639009    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.639051    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.639190    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.640247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.639278    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.678767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.678733    5065 scope.go:122] "RemoveContainer" containerID="786f361b082082eec20679b101290fea6b791d121a70427f51e9c4bbde692964"
Oct 30 18:51:01.759347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.759307    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.762779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.762750    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.763036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.763018    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.763553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.763531    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.763804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.763786    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.768432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.768374    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.778211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.778186    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5bp\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.778673 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.778654    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.778825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.778812    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.764195    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.761475    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.781464    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.763465    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.764405    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.809111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.806957    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.835314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.830958    5065 scope.go:122] "RemoveContainer" containerID="4220d6c836244db887c7a7944686c811b8d01303637054720161f0634e8ef8ba"
Oct 30 18:51:01.835314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.833242    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5bp\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.837889 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.837810    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:01.842657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:01.842595    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\") pod \"e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") " pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:02.117429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.117387    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:02.421439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.421412    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:51:02.425169 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:02.425136    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3252a0df_cb7e_49ec_b5cb_330415724228.slice/crio-d9d1142977c48fe3dfd500237eb3db8bf3bdf16843bd782b8c62d80c2a30b980 WatchSource:0}: Error finding container d9d1142977c48fe3dfd500237eb3db8bf3bdf16843bd782b8c62d80c2a30b980: Status 404 returned error can't find the container with id d9d1142977c48fe3dfd500237eb3db8bf3bdf16843bd782b8c62d80c2a30b980
Oct 30 18:51:02.427677 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.427646    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:51:02.427795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.427726    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:51:02.427795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.427759    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:51:02.674676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.674595    5065 patch_prober.go:29] interesting pod/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:51:02.674676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:02.674650    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:51:03.007155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:03.007041    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerStarted","Data":"1b8817c90a28993438df22460dc52be021f6f224c4fb333315ad5c19db73e79f"}
Oct 30 18:51:03.007337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:03.007165    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerStarted","Data":"d9d1142977c48fe3dfd500237eb3db8bf3bdf16843bd782b8c62d80c2a30b980"}
Oct 30 18:51:03.047342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:03.044794    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podStartSLOduration=2.044742907 podStartE2EDuration="2.044742907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:03.037380548 +0000 UTC m=+1083.300845526" watchObservedRunningTime="2026-10-30 18:51:03.044742907 +0000 UTC m=+1083.308207858"
Oct 30 18:51:03.117728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:03.117695    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:03.124493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:03.124465    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:04.018848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:04.018815    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:04.019838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:04.019805    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:51:06.592452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.592425    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:51:06.708417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708387    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708511    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708556    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708592    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708639    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708673    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:06.708749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708681    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.708749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708721    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.709003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708784    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\") pod \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\" (UID: \"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e\") "
Oct 30 18:51:06.709003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.708807    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:06.709139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709017    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:06.709235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709207    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709362    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709660    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709680    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709689    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709698    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.709725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.709706    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.711901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.711870    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:51:06.712002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.711941    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:06.712002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.711990    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:06.712218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.712053    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d" pod "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" (UID: "861f1a43-bc2b-4528-8f8b-0a88fa03cb8e"). InnerVolumeSpecName "kube-api-access-jdf8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:06.811916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.811854    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.811916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.811886    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.811916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.811902    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdf8d\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-kube-api-access-jdf8d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:06.812117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:06.811917    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:07.055598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.055569    5065 generic.go:356] "Generic (PLEG): container finished" podID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerID="9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e" exitCode=0
Oct 30 18:51:07.055792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.055623    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"
Oct 30 18:51:07.055792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.055641    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerDied","Data":"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"}
Oct 30 18:51:07.055792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.055682    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8" event={"ID":"861f1a43-bc2b-4528-8f8b-0a88fa03cb8e","Type":"ContainerDied","Data":"762be95533e4d6a58597c9fd7e3d133829328742b6a3ba26bea9224956cb88fa"}
Oct 30 18:51:07.055792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.055710    5065 scope.go:122] "RemoveContainer" containerID="9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"
Oct 30 18:51:07.069272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.069247    5065 scope.go:122] "RemoveContainer" containerID="8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"
Oct 30 18:51:07.087796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.087312    5065 scope.go:122] "RemoveContainer" containerID="9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"
Oct 30 18:51:07.087796 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:07.087640    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e\": container with ID starting with 9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e not found: ID does not exist" containerID="9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"
Oct 30 18:51:07.087796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.087675    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e"} err="failed to get container status \"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e\": rpc error: code = NotFound desc = could not find container \"9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e\": container with ID starting with 9d0da3e4ef15e1a00250f23b38afc17cbbffa032abecf6f60c120eaca08d165e not found: ID does not exist"
Oct 30 18:51:07.087796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.087707    5065 scope.go:122] "RemoveContainer" containerID="8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"
Oct 30 18:51:07.088111 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:07.087970    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74\": container with ID starting with 8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74 not found: ID does not exist" containerID="8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"
Oct 30 18:51:07.088111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.087993    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74"} err="failed to get container status \"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74\": rpc error: code = NotFound desc = could not find container \"8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74\": container with ID starting with 8da21c02c3afe2fb190494a0ed5fb9a93f5984364d33063503e236e745883d74 not found: ID does not exist"
Oct 30 18:51:07.112504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.112473    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:51:07.120921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.120893    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f9521e-openshift-default-54d8bc879-pb9q8"]
Oct 30 18:51:07.218454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.218422    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:07.219135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219117    5065 cpu_manager.go:414] "containerMap: removing container" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.219135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219136    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.219301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219161    5065 cpu_manager.go:414] "containerMap: removing container" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.219301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219168    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.219400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219385    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.219459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.219404    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" containerName="istio-proxy"
Oct 30 18:51:07.224658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.224635    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.237845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.237074    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-52fa8b-dockercfg-564k5\""
Oct 30 18:51:07.237845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.237185    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-52fa8b\""
Oct 30 18:51:07.250724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.250692    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:07.275514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.275487    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:07.287058 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.287031    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.287988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.287961    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:07.320015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.319938    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rl8\" (UniqueName: \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.320015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.319990    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.320224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.320044    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8whp\" (UniqueName: \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.320300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.320278    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.423254    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.423440    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rl8\" (UniqueName: \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.423481    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.423529    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8whp\" (UniqueName: \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.424905    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.429166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.426059    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.471374 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.470780    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rl8\" (UniqueName: \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.479117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.474703    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8whp\" (UniqueName: \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\") pod \"payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.575198 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.569672    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:07.605244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.605212    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:07.902270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.902182    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861f1a43-bc2b-4528-8f8b-0a88fa03cb8e" path="/var/lib/kubelet/pods/861f1a43-bc2b-4528-8f8b-0a88fa03cb8e/volumes"
Oct 30 18:51:07.939755 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:07.939687    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf989de23_dd07_453a_924e_f3db35162612.slice/crio-0289084bbc705e165e8a6fa8af31bb9cb0065ecb020e7ac5167885dc313f7b17 WatchSource:0}: Error finding container 0289084bbc705e165e8a6fa8af31bb9cb0065ecb020e7ac5167885dc313f7b17: Status 404 returned error can't find the container with id 0289084bbc705e165e8a6fa8af31bb9cb0065ecb020e7ac5167885dc313f7b17
Oct 30 18:51:07.942913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.942414    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:07.948122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:07.947647    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:07.949365 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:07.949341    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c087ad7_8a4e_40ff_8646_70773abeb054.slice/crio-c3de79c99f342375857938ef7bd993de66bdd33c5e5dc820bfb3f3ed7f6c47d5 WatchSource:0}: Error finding container c3de79c99f342375857938ef7bd993de66bdd33c5e5dc820bfb3f3ed7f6c47d5: Status 404 returned error can't find the container with id c3de79c99f342375857938ef7bd993de66bdd33c5e5dc820bfb3f3ed7f6c47d5
Oct 30 18:51:08.065777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.065747    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" event={"ID":"f989de23-dd07-453a-924e-f3db35162612","Type":"ContainerStarted","Data":"0289084bbc705e165e8a6fa8af31bb9cb0065ecb020e7ac5167885dc313f7b17"}
Oct 30 18:51:08.067592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.067566    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" event={"ID":"5c087ad7-8a4e-40ff-8646-70773abeb054","Type":"ContainerStarted","Data":"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"}
Oct 30 18:51:08.067715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.067609    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" event={"ID":"5c087ad7-8a4e-40ff-8646-70773abeb054","Type":"ContainerStarted","Data":"c3de79c99f342375857938ef7bd993de66bdd33c5e5dc820bfb3f3ed7f6c47d5"}
Oct 30 18:51:08.067715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.067630    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:08.096340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.096281    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" podStartSLOduration=1.09626545 podStartE2EDuration="1.09626545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.09567984 +0000 UTC m=+1088.359144789" watchObservedRunningTime="2026-10-30 18:51:08.09626545 +0000 UTC m=+1088.359730401"
Oct 30 18:51:08.112229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.112206    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:08.117724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.117706    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.128929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.128563    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:08.133426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.133406    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0d540b\""
Oct 30 18:51:08.133531 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.133479    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0d540b-dockercfg-lxf6j\""
Oct 30 18:51:08.150499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.150474    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:08.155365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.155317    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.168262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.168235    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:08.279119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.275162    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsdx\" (UniqueName: \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.279119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.275537    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.279119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.275634    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpzz\" (UniqueName: \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.279119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.275662    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.393314    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsdx\" (UniqueName: \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.393590    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.393678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpzz\" (UniqueName: \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.393706    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.394591    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.394775    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.436166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.431309    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpzz\" (UniqueName: \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.444633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.444362    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:08.445693 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.445669    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsdx\" (UniqueName: \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\") pod \"payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.472037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.472003    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:08.801511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.801460    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:08.812875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:08.812847    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:09.037015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.036985    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:09.041728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.041711    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.055184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.055050    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5ad713\""
Oct 30 18:51:09.056988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.056965    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:09.106603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.106568    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.106793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.106775    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9tg\" (UniqueName: \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.108739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.108700    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" event={"ID":"ee134549-40b2-4218-890d-3dac4d1601d2","Type":"ContainerStarted","Data":"81cd60209e378323af2838155c24c35fa12acc1c48d6d9f8355aa2cfc3574c90"}
Oct 30 18:51:09.108860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.108762    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" event={"ID":"ee134549-40b2-4218-890d-3dac4d1601d2","Type":"ContainerStarted","Data":"cb2ee293ba5fd1cf27fab5ac5d4d62e30c98c0ec29613c89a006db1d7122103b"}
Oct 30 18:51:09.109046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.109024    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:09.111253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.111227    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" event={"ID":"15f0d275-18cb-427b-817c-c7c3dfb80df7","Type":"ContainerStarted","Data":"615430b8b54cf08e4f23dfd444e4ae6ff288e13b4ae5540c6669e9b3d0837931"}
Oct 30 18:51:09.111386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.111278    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" event={"ID":"15f0d275-18cb-427b-817c-c7c3dfb80df7","Type":"ContainerStarted","Data":"ffb8b1e82c48b5154d33c80a815db2a5c82361914f5d0758f1de365616b2340d"}
Oct 30 18:51:09.111386 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.111309    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:09.114340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.114317    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" event={"ID":"f989de23-dd07-453a-924e-f3db35162612","Type":"ContainerStarted","Data":"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"}
Oct 30 18:51:09.114444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.114376    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:09.133873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.133826    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" podStartSLOduration=1.133813982 podStartE2EDuration="1.133813982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:09.130056162 +0000 UTC m=+1089.393521134" watchObservedRunningTime="2026-10-30 18:51:09.133813982 +0000 UTC m=+1089.397278930"
Oct 30 18:51:09.156140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.156096    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" podStartSLOduration=1.156069725 podStartE2EDuration="1.156069725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:09.152280392 +0000 UTC m=+1089.415745365" watchObservedRunningTime="2026-10-30 18:51:09.156069725 +0000 UTC m=+1089.419534672"
Oct 30 18:51:09.178620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.178576    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" podStartSLOduration=2.178563105 podStartE2EDuration="2.178563105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:09.174806106 +0000 UTC m=+1089.438271055" watchObservedRunningTime="2026-10-30 18:51:09.178563105 +0000 UTC m=+1089.442028052"
Oct 30 18:51:09.208637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.208607    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.208846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.208787    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9tg\" (UniqueName: \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.209663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.209638    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.222571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.222547    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9tg\" (UniqueName: \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\") pod \"payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.362537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.359299    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:09.848009 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:09.847976    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53acd10_3e71_40ab_85df_7d83bd3b555b.slice/crio-48cae8fb3ee34dae52a2470e9cddd3fb3cad45a0f905611d5e1ce060830b25af WatchSource:0}: Error finding container 48cae8fb3ee34dae52a2470e9cddd3fb3cad45a0f905611d5e1ce060830b25af: Status 404 returned error can't find the container with id 48cae8fb3ee34dae52a2470e9cddd3fb3cad45a0f905611d5e1ce060830b25af
Oct 30 18:51:09.848599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:09.848529    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:10.125182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:10.125150    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" event={"ID":"c53acd10-3e71-40ab-85df-7d83bd3b555b","Type":"ContainerStarted","Data":"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"}
Oct 30 18:51:10.125368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:10.125206    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" event={"ID":"c53acd10-3e71-40ab-85df-7d83bd3b555b","Type":"ContainerStarted","Data":"48cae8fb3ee34dae52a2470e9cddd3fb3cad45a0f905611d5e1ce060830b25af"}
Oct 30 18:51:10.125687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:10.125664    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:10.153137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:10.153075    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" podStartSLOduration=1.153061776 podStartE2EDuration="1.153061776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:10.150272374 +0000 UTC m=+1090.413737340" watchObservedRunningTime="2026-10-30 18:51:10.153061776 +0000 UTC m=+1090.416526724"
Oct 30 18:51:19.117378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:19.117347    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:20.129024 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:20.128992    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:20.129541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:20.129056    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:20.129541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:20.129109    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:22.150265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:22.150238    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:23.102059 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:23.099692    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:51:24.424158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.424116    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"]
Oct 30 18:51:24.430142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.430119    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.440144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.438680    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:24.440144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.438776    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:51:24.442184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.442156    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"]
Oct 30 18:51:24.444226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.444202    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.444553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.444531    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.444856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.444836    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85vrm\" (UniqueName: \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.547150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.547078    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.548502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.548474    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85vrm\" (UniqueName: \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.548805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.548785    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.557263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.557239    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.558697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.558578    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.590146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.586731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85vrm\" (UniqueName: \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\") pod \"maas-api-revoke-keys-e2e-mt-2aada387-xgmtf\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:24.841721 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:24.841691    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:25.169238 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:25.169206    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bd2554_a05f_43a5_b962_0a851bc2cc59.slice/crio-4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8 WatchSource:0}: Error finding container 4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8: Status 404 returned error can't find the container with id 4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8
Oct 30 18:51:25.170629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:25.170599    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"]
Oct 30 18:51:25.367630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:25.367590    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf" event={"ID":"79bd2554-a05f-43a5-b962-0a851bc2cc59","Type":"ContainerStarted","Data":"4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8"}
Oct 30 18:51:25.630820 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:25.630779    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:25.646197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:25.645616    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:26.254109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:26.253405    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:26.315269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:26.314804    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:26.399796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:26.397902    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf" event={"ID":"79bd2554-a05f-43a5-b962-0a851bc2cc59","Type":"ContainerStarted","Data":"a487aab2eafae39991567451646fe61124b3398abe75932f9a60dc6be4fd1077"}
Oct 30 18:51:27.410489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.410451    5065 generic.go:356] "Generic (PLEG): container finished" podID="79bd2554-a05f-43a5-b962-0a851bc2cc59" containerID="a487aab2eafae39991567451646fe61124b3398abe75932f9a60dc6be4fd1077" exitCode=0
Oct 30 18:51:27.410944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.410539    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf" event={"ID":"79bd2554-a05f-43a5-b962-0a851bc2cc59","Type":"ContainerDied","Data":"a487aab2eafae39991567451646fe61124b3398abe75932f9a60dc6be4fd1077"}
Oct 30 18:51:27.617236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.617212    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:27.648612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.648585    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\" (UniqueName: \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\") pod \"79bd2554-a05f-43a5-b962-0a851bc2cc59\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") "
Oct 30 18:51:27.648813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.648775    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\") pod \"79bd2554-a05f-43a5-b962-0a851bc2cc59\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") "
Oct 30 18:51:27.648888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.648842    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\") pod \"79bd2554-a05f-43a5-b962-0a851bc2cc59\" (UID: \"79bd2554-a05f-43a5-b962-0a851bc2cc59\") "
Oct 30 18:51:27.649456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.649429    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca" pod "79bd2554-a05f-43a5-b962-0a851bc2cc59" (UID: "79bd2554-a05f-43a5-b962-0a851bc2cc59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:27.649808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.649785    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:27.651861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.651834    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca" pod "79bd2554-a05f-43a5-b962-0a851bc2cc59" (UID: "79bd2554-a05f-43a5-b962-0a851bc2cc59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:27.652030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.652006    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm" pod "79bd2554-a05f-43a5-b962-0a851bc2cc59" (UID: "79bd2554-a05f-43a5-b962-0a851bc2cc59"). InnerVolumeSpecName "kube-api-access-85vrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:27.751579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.751546    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bd2554-a05f-43a5-b962-0a851bc2cc59-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:27.751579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:27.751576    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85vrm\" (UniqueName: \"kubernetes.io/projected/79bd2554-a05f-43a5-b962-0a851bc2cc59-kube-api-access-85vrm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:28.424344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:28.424308    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"
Oct 30 18:51:28.424772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:28.424316    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf" event={"ID":"79bd2554-a05f-43a5-b962-0a851bc2cc59","Type":"ContainerDied","Data":"4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8"}
Oct 30 18:51:28.424772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:28.424484    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5eb421776b7fdaa5c876751b3ef38989d3c632eadfa7140383ef5f69e9d1d8"
Oct 30 18:51:30.631930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:30.631876    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" podUID="5c087ad7-8a4e-40ff-8646-70773abeb054" containerName="payload-processing" containerID="cri-o://f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb" gracePeriod=25
Oct 30 18:51:30.647037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:30.647005    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" podUID="f989de23-dd07-453a-924e-f3db35162612" containerName="payload-pre-processing" containerID="cri-o://46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80" gracePeriod=25
Oct 30 18:51:31.016138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.010745    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:31.090163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.090136    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:31.128006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.127972    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\") pod \"f989de23-dd07-453a-924e-f3db35162612\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") "
Oct 30 18:51:31.128191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.128078    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\" (UniqueName: \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\") pod \"f989de23-dd07-453a-924e-f3db35162612\" (UID: \"f989de23-dd07-453a-924e-f3db35162612\") "
Oct 30 18:51:31.129602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.129503    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume" pod "f989de23-dd07-453a-924e-f3db35162612" (UID: "f989de23-dd07-453a-924e-f3db35162612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:31.131167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.131139    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8" pod "f989de23-dd07-453a-924e-f3db35162612" (UID: "f989de23-dd07-453a-924e-f3db35162612"). InnerVolumeSpecName "kube-api-access-45rl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:31.230741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.230663    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\") pod \"5c087ad7-8a4e-40ff-8646-70773abeb054\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") "
Oct 30 18:51:31.230741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.230703    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\" (UniqueName: \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\") pod \"5c087ad7-8a4e-40ff-8646-70773abeb054\" (UID: \"5c087ad7-8a4e-40ff-8646-70773abeb054\") "
Oct 30 18:51:31.231155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.231123    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume" pod "5c087ad7-8a4e-40ff-8646-70773abeb054" (UID: "5c087ad7-8a4e-40ff-8646-70773abeb054"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:31.231298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.231265    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f989de23-dd07-453a-924e-f3db35162612-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.231298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.231279    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rl8\" (UniqueName: \"kubernetes.io/projected/f989de23-dd07-453a-924e-f3db35162612-kube-api-access-45rl8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.231298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.231288    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c087ad7-8a4e-40ff-8646-70773abeb054-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.232955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.232932    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp" pod "5c087ad7-8a4e-40ff-8646-70773abeb054" (UID: "5c087ad7-8a4e-40ff-8646-70773abeb054"). InnerVolumeSpecName "kube-api-access-k8whp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:31.257682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.257653    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" podUID="15f0d275-18cb-427b-817c-c7c3dfb80df7" containerName="payload-processing" containerID="cri-o://615430b8b54cf08e4f23dfd444e4ae6ff288e13b4ae5540c6669e9b3d0837931" gracePeriod=25
Oct 30 18:51:31.315541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.315506    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" podUID="ee134549-40b2-4218-890d-3dac4d1601d2" containerName="payload-pre-processing" containerID="cri-o://81cd60209e378323af2838155c24c35fa12acc1c48d6d9f8355aa2cfc3574c90" gracePeriod=25
Oct 30 18:51:31.332850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.332826    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8whp\" (UniqueName: \"kubernetes.io/projected/5c087ad7-8a4e-40ff-8646-70773abeb054-kube-api-access-k8whp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.453171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.453008    5065 generic.go:356] "Generic (PLEG): container finished" podID="ee134549-40b2-4218-890d-3dac4d1601d2" containerID="81cd60209e378323af2838155c24c35fa12acc1c48d6d9f8355aa2cfc3574c90" exitCode=0
Oct 30 18:51:31.453171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.453045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" event={"ID":"ee134549-40b2-4218-890d-3dac4d1601d2","Type":"ContainerDied","Data":"81cd60209e378323af2838155c24c35fa12acc1c48d6d9f8355aa2cfc3574c90"}
Oct 30 18:51:31.455338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.455307    5065 generic.go:356] "Generic (PLEG): container finished" podID="15f0d275-18cb-427b-817c-c7c3dfb80df7" containerID="615430b8b54cf08e4f23dfd444e4ae6ff288e13b4ae5540c6669e9b3d0837931" exitCode=0
Oct 30 18:51:31.455463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.455382    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" event={"ID":"15f0d275-18cb-427b-817c-c7c3dfb80df7","Type":"ContainerDied","Data":"615430b8b54cf08e4f23dfd444e4ae6ff288e13b4ae5540c6669e9b3d0837931"}
Oct 30 18:51:31.457263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.457234    5065 generic.go:356] "Generic (PLEG): container finished" podID="f989de23-dd07-453a-924e-f3db35162612" containerID="46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80" exitCode=0
Oct 30 18:51:31.457365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.457296    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"
Oct 30 18:51:31.457365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.457302    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" event={"ID":"f989de23-dd07-453a-924e-f3db35162612","Type":"ContainerDied","Data":"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"}
Oct 30 18:51:31.457365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.457344    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc" event={"ID":"f989de23-dd07-453a-924e-f3db35162612","Type":"ContainerDied","Data":"0289084bbc705e165e8a6fa8af31bb9cb0065ecb020e7ac5167885dc313f7b17"}
Oct 30 18:51:31.457518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.457387    5065 scope.go:122] "RemoveContainer" containerID="46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"
Oct 30 18:51:31.461297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.461279    5065 generic.go:356] "Generic (PLEG): container finished" podID="5c087ad7-8a4e-40ff-8646-70773abeb054" containerID="f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb" exitCode=0
Oct 30 18:51:31.461405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.461392    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" event={"ID":"5c087ad7-8a4e-40ff-8646-70773abeb054","Type":"ContainerDied","Data":"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"}
Oct 30 18:51:31.461486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.461423    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn" event={"ID":"5c087ad7-8a4e-40ff-8646-70773abeb054","Type":"ContainerDied","Data":"c3de79c99f342375857938ef7bd993de66bdd33c5e5dc820bfb3f3ed7f6c47d5"}
Oct 30 18:51:31.461486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.461392    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"
Oct 30 18:51:31.492008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.491982    5065 scope.go:122] "RemoveContainer" containerID="46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"
Oct 30 18:51:31.497396 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:31.493640    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80\": container with ID starting with 46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80 not found: ID does not exist" containerID="46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"
Oct 30 18:51:31.497396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.493694    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80"} err="failed to get container status \"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80\": rpc error: code = NotFound desc = could not find container \"46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80\": container with ID starting with 46df10388b70d786d217406d5c2731120fefbc24c94ab3ac03a5dabba90c3e80 not found: ID does not exist"
Oct 30 18:51:31.497396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.493723    5065 scope.go:122] "RemoveContainer" containerID="f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"
Oct 30 18:51:31.544571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.544541    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:31.548927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.548902    5065 scope.go:122] "RemoveContainer" containerID="f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"
Oct 30 18:51:31.549363 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:31.549329    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb\": container with ID starting with f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb not found: ID does not exist" containerID="f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"
Oct 30 18:51:31.549495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.549372    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb"} err="failed to get container status \"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb\": rpc error: code = NotFound desc = could not find container \"f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb\": container with ID starting with f402b0864642eabe95c55372ee5268552f37c774146517e757f6bcae111506fb not found: ID does not exist"
Oct 30 18:51:31.550680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.550627    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-c5c7cc564-g5pnn"]
Oct 30 18:51:31.558490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.558467    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:31.565989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.565968    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-d849bc69f-sjkrc"]
Oct 30 18:51:31.617920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.617890    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:31.765478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.760159    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\") pod \"15f0d275-18cb-427b-817c-c7c3dfb80df7\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") "
Oct 30 18:51:31.765478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.760269    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\" (UniqueName: \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\") pod \"15f0d275-18cb-427b-817c-c7c3dfb80df7\" (UID: \"15f0d275-18cb-427b-817c-c7c3dfb80df7\") "
Oct 30 18:51:31.765478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.763498    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx" pod "15f0d275-18cb-427b-817c-c7c3dfb80df7" (UID: "15f0d275-18cb-427b-817c-c7c3dfb80df7"). InnerVolumeSpecName "kube-api-access-rbsdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:31.765478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.763848    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume" pod "15f0d275-18cb-427b-817c-c7c3dfb80df7" (UID: "15f0d275-18cb-427b-817c-c7c3dfb80df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:31.800542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.800519    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:31.862516 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.862491    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f0d275-18cb-427b-817c-c7c3dfb80df7-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.862516 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.862518    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbsdx\" (UniqueName: \"kubernetes.io/projected/15f0d275-18cb-427b-817c-c7c3dfb80df7-kube-api-access-rbsdx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.892348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.892315    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c087ad7-8a4e-40ff-8646-70773abeb054" path="/var/lib/kubelet/pods/5c087ad7-8a4e-40ff-8646-70773abeb054/volumes"
Oct 30 18:51:31.892757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.892742    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f989de23-dd07-453a-924e-f3db35162612" path="/var/lib/kubelet/pods/f989de23-dd07-453a-924e-f3db35162612/volumes"
Oct 30 18:51:31.963363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.963339    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\") pod \"ee134549-40b2-4218-890d-3dac4d1601d2\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") "
Oct 30 18:51:31.963544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.963532    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\" (UniqueName: \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\") pod \"ee134549-40b2-4218-890d-3dac4d1601d2\" (UID: \"ee134549-40b2-4218-890d-3dac4d1601d2\") "
Oct 30 18:51:31.963728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.963707    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume" pod "ee134549-40b2-4218-890d-3dac4d1601d2" (UID: "ee134549-40b2-4218-890d-3dac4d1601d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:31.963993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.963973    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee134549-40b2-4218-890d-3dac4d1601d2-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.965680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:31.965659    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz" pod "ee134549-40b2-4218-890d-3dac4d1601d2" (UID: "ee134549-40b2-4218-890d-3dac4d1601d2"). InnerVolumeSpecName "kube-api-access-tnpzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:32.065626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.065557    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpzz\" (UniqueName: \"kubernetes.io/projected/ee134549-40b2-4218-890d-3dac4d1601d2-kube-api-access-tnpzz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.478335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.478245    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"
Oct 30 18:51:32.478517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.478243    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7" event={"ID":"ee134549-40b2-4218-890d-3dac4d1601d2","Type":"ContainerDied","Data":"cb2ee293ba5fd1cf27fab5ac5d4d62e30c98c0ec29613c89a006db1d7122103b"}
Oct 30 18:51:32.478517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.478399    5065 scope.go:122] "RemoveContainer" containerID="81cd60209e378323af2838155c24c35fa12acc1c48d6d9f8355aa2cfc3574c90"
Oct 30 18:51:32.480546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.480514    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j" event={"ID":"15f0d275-18cb-427b-817c-c7c3dfb80df7","Type":"ContainerDied","Data":"ffb8b1e82c48b5154d33c80a815db2a5c82361914f5d0758f1de365616b2340d"}
Oct 30 18:51:32.480664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.480628    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"
Oct 30 18:51:32.505944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.505634    5065 scope.go:122] "RemoveContainer" containerID="615430b8b54cf08e4f23dfd444e4ae6ff288e13b4ae5540c6669e9b3d0837931"
Oct 30 18:51:32.513901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.513873    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:32.522276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.522246    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-5fd6c9d847-npqv7"]
Oct 30 18:51:32.541382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.541345    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:32.547582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.547556    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-67cb4c54bb-8zn6j"]
Oct 30 18:51:32.804728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:32.804694    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:33.143390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.143321    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:51:33.144022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144005    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ee134549-40b2-4218-890d-3dac4d1601d2" containerName="payload-pre-processing"
Oct 30 18:51:33.144074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144023    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee134549-40b2-4218-890d-3dac4d1601d2" containerName="payload-pre-processing"
Oct 30 18:51:33.144074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144042    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5c087ad7-8a4e-40ff-8646-70773abeb054" containerName="payload-processing"
Oct 30 18:51:33.144074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144048    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c087ad7-8a4e-40ff-8646-70773abeb054" containerName="payload-processing"
Oct 30 18:51:33.144074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144054    5065 cpu_manager.go:414] "containerMap: removing container" podUID="79bd2554-a05f-43a5-b962-0a851bc2cc59" containerName="revoke-keys"
Oct 30 18:51:33.144074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144059    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bd2554-a05f-43a5-b962-0a851bc2cc59" containerName="revoke-keys"
Oct 30 18:51:33.144236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144109    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f989de23-dd07-453a-924e-f3db35162612" containerName="payload-pre-processing"
Oct 30 18:51:33.144236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144117    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f989de23-dd07-453a-924e-f3db35162612" containerName="payload-pre-processing"
Oct 30 18:51:33.144236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144133    5065 cpu_manager.go:414] "containerMap: removing container" podUID="15f0d275-18cb-427b-817c-c7c3dfb80df7" containerName="payload-processing"
Oct 30 18:51:33.144236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144141    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f0d275-18cb-427b-817c-c7c3dfb80df7" containerName="payload-processing"
Oct 30 18:51:33.144352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144295    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f989de23-dd07-453a-924e-f3db35162612" containerName="payload-pre-processing"
Oct 30 18:51:33.144352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144305    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f0d275-18cb-427b-817c-c7c3dfb80df7" containerName="payload-processing"
Oct 30 18:51:33.144352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144316    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee134549-40b2-4218-890d-3dac4d1601d2" containerName="payload-pre-processing"
Oct 30 18:51:33.144352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144324    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c087ad7-8a4e-40ff-8646-70773abeb054" containerName="payload-processing"
Oct 30 18:51:33.144352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.144329    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bd2554-a05f-43a5-b962-0a851bc2cc59" containerName="revoke-keys"
Oct 30 18:51:33.149714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.149694    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.158582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.158560    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-52fa8b\""
Oct 30 18:51:33.158582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.158561    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-52fa8b-dockercfg-5vmpk\""
Oct 30 18:51:33.170493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.170469    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:51:33.207862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.207819    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:51:33.213602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.213570    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.227806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.227777    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:51:33.307234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.302914    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskp9\" (UniqueName: \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.334116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.328922    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.440184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.434019    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5xf\" (UniqueName: \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.440184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.434179    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.440184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.434255    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskp9\" (UniqueName: \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.440184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.434297    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.440184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.435435    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.474886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.474849    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskp9\" (UniqueName: \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\") pod \"payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.535713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.535667    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.535954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.535933    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5xf\" (UniqueName: \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.536480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.536457    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.549933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.549907    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5xf\" (UniqueName: \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\") pod \"payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.778524 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.778241    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:33.837529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.837497    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:33.893108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.892525    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f0d275-18cb-427b-817c-c7c3dfb80df7" path="/var/lib/kubelet/pods/15f0d275-18cb-427b-817c-c7c3dfb80df7/volumes"
Oct 30 18:51:33.893108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:33.893064    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee134549-40b2-4218-890d-3dac4d1601d2" path="/var/lib/kubelet/pods/ee134549-40b2-4218-890d-3dac4d1601d2/volumes"
Oct 30 18:51:34.137507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.137485    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:51:34.140490 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:34.140465    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0767d5_b349_45cf_9f6c_47ad7bcadd9c.slice/crio-40a64ad83eca835abc3dc14f6bd83c4d476f4dc7a849d3205d9c91fb2f3e499d WatchSource:0}: Error finding container 40a64ad83eca835abc3dc14f6bd83c4d476f4dc7a849d3205d9c91fb2f3e499d: Status 404 returned error can't find the container with id 40a64ad83eca835abc3dc14f6bd83c4d476f4dc7a849d3205d9c91fb2f3e499d
Oct 30 18:51:34.142552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.142530    5065 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:51:34.153375 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:34.153353    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5df24f1_8904_4d57_9874_bea45240fa86.slice/crio-2c12eb602efd38fe2995d7848678432453917b53782bb740539410e1e18c8ac3 WatchSource:0}: Error finding container 2c12eb602efd38fe2995d7848678432453917b53782bb740539410e1e18c8ac3: Status 404 returned error can't find the container with id 2c12eb602efd38fe2995d7848678432453917b53782bb740539410e1e18c8ac3
Oct 30 18:51:34.158139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.158111    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:51:34.518231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.518189    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" event={"ID":"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c","Type":"ContainerStarted","Data":"40a64ad83eca835abc3dc14f6bd83c4d476f4dc7a849d3205d9c91fb2f3e499d"}
Oct 30 18:51:34.519684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.519663    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" event={"ID":"b5df24f1-8904-4d57-9874-bea45240fa86","Type":"ContainerStarted","Data":"2c12eb602efd38fe2995d7848678432453917b53782bb740539410e1e18c8ac3"}
Oct 30 18:51:34.699110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.698961    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:51:34.719404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.712875    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:34.737047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.737015    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:51:34.738618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.738591    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0d540b\""
Oct 30 18:51:34.745214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.745130    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0d540b-dockercfg-zb2xf\""
Oct 30 18:51:34.769442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.769366    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:51:34.800426 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.800399    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:34.839180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.821315    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:51:34.864653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.864614    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nqh\" (UniqueName: \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:34.864846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.864679    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:34.966449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.966412    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfnt\" (UniqueName: \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:34.966843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.966817    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nqh\" (UniqueName: \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:34.966962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.966864    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:34.967033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.966964    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:34.967829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.967800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:35.002898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:34.999165    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nqh\" (UniqueName: \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\") pod \"payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:35.049246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.049166    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:35.071591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.071562    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:35.071985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.071692    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfnt\" (UniqueName: \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:35.072847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.072826    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:35.096237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.096212    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfnt\" (UniqueName: \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\") pod \"payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:35.164128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.156363    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:35.543535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.543445    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:51:35.599249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.599220    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" event={"ID":"3851f70c-26ab-4766-a9e5-c3080e67fd81","Type":"ContainerStarted","Data":"0afe6947966724edf5a8f6867147f76dcdd113c46a4a13bd4405fe980f526938"}
Oct 30 18:51:35.670641 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:35.670602    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a525d3_325f_4b1e_a324_591efaf9cd78.slice/crio-6be04496ac8eebf94aa8dcec9e36138ba03cc926873badc8f7aa69d3523fbb54 WatchSource:0}: Error finding container 6be04496ac8eebf94aa8dcec9e36138ba03cc926873badc8f7aa69d3523fbb54: Status 404 returned error can't find the container with id 6be04496ac8eebf94aa8dcec9e36138ba03cc926873badc8f7aa69d3523fbb54
Oct 30 18:51:35.726356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:35.726322    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:51:36.633239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:36.633206    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" event={"ID":"71a525d3-325f-4b1e-a324-591efaf9cd78","Type":"ContainerStarted","Data":"6be04496ac8eebf94aa8dcec9e36138ba03cc926873badc8f7aa69d3523fbb54"}
Oct 30 18:51:37.105159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.101352    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:51:37.107457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.107432    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.113465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.113433    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5ad713-dockercfg-7w5kx\""
Oct 30 18:51:37.170041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.170003    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:51:37.355561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.355468    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr97z\" (UniqueName: \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.355561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.355549    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.392683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.392636    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:51:37.400791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.400765    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.423942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.423905    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:51:37.457014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.456976    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfhr\" (UniqueName: \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.457493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.457467    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr97z\" (UniqueName: \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.457681 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.457665    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.457964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.457876    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.459015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.458988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.474609 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.474560    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr97z\" (UniqueName: \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.572474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.569761    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.572474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.570147    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfhr\" (UniqueName: \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.572474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.571425    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.593397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.593322    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfhr\" (UniqueName: \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\") pod \"payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.690383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.690286    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:37.735689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.735660    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:37.805856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:37.805598    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" podUID="c53acd10-3e71-40ab-85df-7d83bd3b555b" containerName="payload-processing" containerID="cri-o://b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294" gracePeriod=25
Oct 30 18:51:38.083764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.083714    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:51:38.098732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.098652    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"]
Oct 30 18:51:38.109466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.109401    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffc6e92-h4t4g"]
Oct 30 18:51:38.160330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.160221    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:51:38.248535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.248504    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:38.292067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.292033    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\" (UniqueName: \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\") pod \"c53acd10-3e71-40ab-85df-7d83bd3b555b\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") "
Oct 30 18:51:38.292664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.292286    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\") pod \"c53acd10-3e71-40ab-85df-7d83bd3b555b\" (UID: \"c53acd10-3e71-40ab-85df-7d83bd3b555b\") "
Oct 30 18:51:38.293203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.293164    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume" pod "c53acd10-3e71-40ab-85df-7d83bd3b555b" (UID: "c53acd10-3e71-40ab-85df-7d83bd3b555b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:38.303167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.303125    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg" pod "c53acd10-3e71-40ab-85df-7d83bd3b555b" (UID: "c53acd10-3e71-40ab-85df-7d83bd3b555b"). InnerVolumeSpecName "kube-api-access-kl9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:38.395595 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.395530    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53acd10-3e71-40ab-85df-7d83bd3b555b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:38.395595 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.395569    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9tg\" (UniqueName: \"kubernetes.io/projected/c53acd10-3e71-40ab-85df-7d83bd3b555b-kube-api-access-kl9tg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:38.682759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.682718    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" event={"ID":"79f32450-fb33-415d-a973-1bb9c1ba9853","Type":"ContainerStarted","Data":"2659ca05706aa98aec11cc4f3cc6efac142099198ba970757f7430bf6b4e4241"}
Oct 30 18:51:38.686481 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.686446    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" event={"ID":"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef","Type":"ContainerStarted","Data":"3a4d898769868fc46fe4edeeec0b8177e27e5e648c99a8bd2a946266c1245392"}
Oct 30 18:51:38.689234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.688746    5065 generic.go:356] "Generic (PLEG): container finished" podID="c53acd10-3e71-40ab-85df-7d83bd3b555b" containerID="b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294" exitCode=0
Oct 30 18:51:38.689234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.689016    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" event={"ID":"c53acd10-3e71-40ab-85df-7d83bd3b555b","Type":"ContainerDied","Data":"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"}
Oct 30 18:51:38.689234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.689061    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz" event={"ID":"c53acd10-3e71-40ab-85df-7d83bd3b555b","Type":"ContainerDied","Data":"48cae8fb3ee34dae52a2470e9cddd3fb3cad45a0f905611d5e1ce060830b25af"}
Oct 30 18:51:38.689234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.689100    5065 scope.go:122] "RemoveContainer" containerID="b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"
Oct 30 18:51:38.689234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.689193    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"
Oct 30 18:51:38.709719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.709543    5065 scope.go:122] "RemoveContainer" containerID="b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"
Oct 30 18:51:38.710993 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:38.710966    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294\": container with ID starting with b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294 not found: ID does not exist" containerID="b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"
Oct 30 18:51:38.711156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.711004    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294"} err="failed to get container status \"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294\": rpc error: code = NotFound desc = could not find container \"b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294\": container with ID starting with b2962f952ef6acf76486575754d50ca34fe81f3dda5b60c3bf00ebd34da10294 not found: ID does not exist"
Oct 30 18:51:38.731500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.731478    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:38.741129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:38.739861    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-69ff6995f8-t6nzz"]
Oct 30 18:51:39.725429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.725340    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" event={"ID":"71a525d3-325f-4b1e-a324-591efaf9cd78","Type":"ContainerStarted","Data":"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"}
Oct 30 18:51:39.726072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.726044    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:39.730320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.730294    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" event={"ID":"b5df24f1-8904-4d57-9874-bea45240fa86","Type":"ContainerStarted","Data":"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"}
Oct 30 18:51:39.730457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.730356    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:39.758061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.758007    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" podStartSLOduration=2.004063643 podStartE2EDuration="5.757988311s" totalImagesPullingTime="3.753924668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:39.750255791 +0000 UTC m=+1120.013720742" watchObservedRunningTime="2026-10-30 18:51:39.757988311 +0000 UTC m=+1120.021453264"
Oct 30 18:51:39.775652 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.775600    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" podStartSLOduration=1.5947347920000001 podStartE2EDuration="6.775587576s" totalImagesPullingTime="5.180852784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:39.773059652 +0000 UTC m=+1120.036524627" watchObservedRunningTime="2026-10-30 18:51:39.775587576 +0000 UTC m=+1120.039052592"
Oct 30 18:51:39.934628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.931605    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fa1054-064e-441a-9e61-d0716bc62a07" path="/var/lib/kubelet/pods/a4fa1054-064e-441a-9e61-d0716bc62a07/volumes"
Oct 30 18:51:39.952014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:39.946908    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53acd10-3e71-40ab-85df-7d83bd3b555b" path="/var/lib/kubelet/pods/c53acd10-3e71-40ab-85df-7d83bd3b555b/volumes"
Oct 30 18:51:40.756025 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.755201    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" event={"ID":"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c","Type":"ContainerStarted","Data":"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"}
Oct 30 18:51:40.756025 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.755344    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:40.761048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.761016    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" event={"ID":"79f32450-fb33-415d-a973-1bb9c1ba9853","Type":"ContainerStarted","Data":"e5208483e3fd9bc402737fc6e6b9b695abef9a074e491a20987a32c3999999e0"}
Oct 30 18:51:40.761861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.761839    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:40.767932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.767907    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" event={"ID":"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef","Type":"ContainerStarted","Data":"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"}
Oct 30 18:51:40.768543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.768521    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:40.770668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.770644    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" event={"ID":"3851f70c-26ab-4766-a9e5-c3080e67fd81","Type":"ContainerStarted","Data":"4fb2fb9bedf3e9cbe8dc021d0d96836bb0486ee555ae62243e13b93c9b2b993c"}
Oct 30 18:51:40.771475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.771142    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:40.787631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.785250    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" podStartSLOduration=1.8744565199999998 podStartE2EDuration="7.785233026s" totalImagesPullingTime="5.910776506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:40.77942945 +0000 UTC m=+1121.042894406" watchObservedRunningTime="2026-10-30 18:51:40.785233026 +0000 UTC m=+1121.048697983"
Oct 30 18:51:40.842281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.842231    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" podStartSLOduration=1.935503744 podStartE2EDuration="3.842214765s" totalImagesPullingTime="1.906711021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:40.838964186 +0000 UTC m=+1121.102429134" watchObservedRunningTime="2026-10-30 18:51:40.842214765 +0000 UTC m=+1121.105679715"
Oct 30 18:51:40.843379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.843343    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" podStartSLOduration=2.138485518 podStartE2EDuration="3.843328112s" totalImagesPullingTime="1.704842594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:40.816392292 +0000 UTC m=+1121.079857244" watchObservedRunningTime="2026-10-30 18:51:40.843328112 +0000 UTC m=+1121.106793061"
Oct 30 18:51:40.867036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:40.866991    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" podStartSLOduration=1.981749886 podStartE2EDuration="6.866975556s" totalImagesPullingTime="4.88522567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:40.861067319 +0000 UTC m=+1121.124532271" watchObservedRunningTime="2026-10-30 18:51:40.866975556 +0000 UTC m=+1121.130440507"
Oct 30 18:51:42.034951 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:42.034920    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"]
Oct 30 18:51:42.040540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:42.040018    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c5fae5-nbx9r"]
Oct 30 18:51:43.895300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:43.895262    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7" path="/var/lib/kubelet/pods/a014dfb8-1c55-4b2e-8348-b8cc4daf4bd7/volumes"
Oct 30 18:51:44.175016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:44.174936    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"]
Oct 30 18:51:44.188848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:44.188817    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2aada387-xgmtf"]
Oct 30 18:51:45.894750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:45.894711    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bd2554-a05f-43a5-b962-0a851bc2cc59" path="/var/lib/kubelet/pods/79bd2554-a05f-43a5-b962-0a851bc2cc59/volumes"
Oct 30 18:51:50.772768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:50.772736    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:51:51.055831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:51.055727    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"]
Oct 30 18:51:51.063459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:51.063428    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d0d074-9gzm5"]
Oct 30 18:51:51.805112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:51.804274    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:51:51.805657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:51.805297    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:51:51.954152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:51.954118    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d" path="/var/lib/kubelet/pods/d8dc6397-f5b2-43ff-9b7a-bf44e5b9977d/volumes"
Oct 30 18:51:52.825804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:52.823247    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:51:52.829858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:52.829836    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:51:52.841236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:52.841211    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:51:53.077888 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:51:53.077810    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:51:54.053583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:54.053547    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"]
Oct 30 18:51:54.073946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:54.073890    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b702625a-2gwp2"]
Oct 30 18:51:55.898707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:55.898674    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecac9af-05e0-43c7-b651-6cd051eccb42" path="/var/lib/kubelet/pods/3ecac9af-05e0-43c7-b651-6cd051eccb42/volumes"
Oct 30 18:51:58.650977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.648935    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:51:58.650977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.650142    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c53acd10-3e71-40ab-85df-7d83bd3b555b" containerName="payload-processing"
Oct 30 18:51:58.650977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.650162    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53acd10-3e71-40ab-85df-7d83bd3b555b" containerName="payload-processing"
Oct 30 18:51:58.650977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.650385    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53acd10-3e71-40ab-85df-7d83bd3b555b" containerName="payload-processing"
Oct 30 18:51:58.680948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.680893    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:51:58.681248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.681010    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.716539 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.716509    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45f649-openshift-default-dockercfg-87plx\""
Oct 30 18:51:58.750964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.750927    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751011    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvnt\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751078    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751143    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751220    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751323    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751351    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751391    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.751480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.751455    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853171    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853276    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853335    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvnt\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853397    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853438    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853515    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853611    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853649    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.853816 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.853689    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.854542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.854508    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.854781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.854519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.855014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.854750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.855913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.855890    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.869543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.869503    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.905157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.901113    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.914109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.906575    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.946552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.946487    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:58.976108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:58.949684    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvnt\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\") pod \"e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") " pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:59.088413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:59.058336    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:51:59.503566 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:51:59.503533    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3d595a_e81f_4461_abd4_4a8174a11c1b.slice/crio-d3b913c2867155e0c42dcb006286c5e045cef39bca8a9a4fdc9988d93240e096 WatchSource:0}: Error finding container d3b913c2867155e0c42dcb006286c5e045cef39bca8a9a4fdc9988d93240e096: Status 404 returned error can't find the container with id d3b913c2867155e0c42dcb006286c5e045cef39bca8a9a4fdc9988d93240e096
Oct 30 18:51:59.505809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:59.505781    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:51:59.507730 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:59.507678    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:51:59.507846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:59.507772    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:51:59.507846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:51:59.507818    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:00.239461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:00.239427    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerStarted","Data":"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"}
Oct 30 18:52:00.239881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:00.239483    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerStarted","Data":"d3b913c2867155e0c42dcb006286c5e045cef39bca8a9a4fdc9988d93240e096"}
Oct 30 18:52:00.281245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:00.273916    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podStartSLOduration=2.273897352 podStartE2EDuration="2.273897352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:00.272911191 +0000 UTC m=+1140.536376140" watchObservedRunningTime="2026-10-30 18:52:00.273897352 +0000 UTC m=+1140.537362303"
Oct 30 18:52:01.074328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.074299    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:01.079758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.079733    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:01.253504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.253463    5065 generic.go:356] "Generic (PLEG): container finished" podID="3252a0df-cb7e-49ec-b5cb-330415724228" containerID="1b8817c90a28993438df22460dc52be021f6f224c4fb333315ad5c19db73e79f" exitCode=0
Oct 30 18:52:01.253920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.253535    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerDied","Data":"1b8817c90a28993438df22460dc52be021f6f224c4fb333315ad5c19db73e79f"}
Oct 30 18:52:01.253920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.253856    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:01.254256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.254233    5065 scope.go:122] "RemoveContainer" containerID="1b8817c90a28993438df22460dc52be021f6f224c4fb333315ad5c19db73e79f"
Oct 30 18:52:01.254675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.254655    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:01.257465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.257431    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:01.257575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.257527    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:01.257619 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.257578    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:01.470605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.470328    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:52:01.490418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.490266    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.501356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.498769    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:52:01.514659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.514635    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9ceefc-openshift-default-dockercfg-7zt9f\""
Oct 30 18:52:01.583981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.583900    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.583981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.583944    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584066    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584206    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584376    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6lv\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584483    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584515    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584540    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.584654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.584607    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691037    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691245    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6lv\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691346    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691379    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691404    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691431    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691514    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691538    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.691616    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.715489    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.716064    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.716750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.716995    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.707285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.717248    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.756717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.746627    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.756717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.753816    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.777109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.772444    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.777109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.773508    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6lv\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\") pod \"e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") " pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:01.850339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:01.850245    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:02.047074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.047036    5065 scope.go:122] "RemoveContainer" containerID="78ea5465bafd5a127692d317439930197dd9c6327372368f50cf966d32c68f92"
Oct 30 18:52:02.070144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.070118    5065 scope.go:122] "RemoveContainer" containerID="0bdec7d8a3d708f9e7674f1de8b5461414f799448fcb72edc2d3dfa4b0bfe674"
Oct 30 18:52:02.081211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.081114    5065 scope.go:122] "RemoveContainer" containerID="fe6cdd7661b4aa889190dcd5a7cb596b4061b51e2f4be7737e3ae530c46eea3a"
Oct 30 18:52:02.265862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.265781    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerStarted","Data":"c712da9fdf340633a883a994b21635edbdd3fcc692d337964ba568df7b369928"}
Oct 30 18:52:02.270266 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:52:02.270243    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ad147b_de5e_42f3_843e_fe309c3384fd.slice/crio-a9e4cf0fbb0de6c07956cac1cf7be49e4c37cb26e18036d75dd198b62fbcf4c2 WatchSource:0}: Error finding container a9e4cf0fbb0de6c07956cac1cf7be49e4c37cb26e18036d75dd198b62fbcf4c2: Status 404 returned error can't find the container with id a9e4cf0fbb0de6c07956cac1cf7be49e4c37cb26e18036d75dd198b62fbcf4c2
Oct 30 18:52:02.272131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.272104    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:52:02.272710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.272682    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:02.272760 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.272742    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:02.272795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:02.272773    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:03.123112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.122770    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:52:03.130124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.124861    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:03.130124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.124938    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:03.278365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.278328    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerStarted","Data":"1a374073e324e792b690251fe0a626cad3a6c0d4491a923cb2a2ad90ff85a53b"}
Oct 30 18:52:03.278859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.278384    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerStarted","Data":"a9e4cf0fbb0de6c07956cac1cf7be49e4c37cb26e18036d75dd198b62fbcf4c2"}
Oct 30 18:52:03.326751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.326701    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podStartSLOduration=2.326686451 podStartE2EDuration="2.326686451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:03.325331428 +0000 UTC m=+1143.588796377" watchObservedRunningTime="2026-10-30 18:52:03.326686451 +0000 UTC m=+1143.590151398"
Oct 30 18:52:03.851707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.851666    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:03.857770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:03.857748    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:04.019207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:04.019128    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:52:04.120210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:04.120077    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:04.120392 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:04.120211    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:04.344505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:04.344475    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:04.358107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:04.354375    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:05.118250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:05.117785    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:05.118250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:05.117856    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:06.118527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:06.118493    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:06.118916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:06.118554    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:07.137469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:07.126222    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:07.137469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:07.126310    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:08.129588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:08.129546    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 18:52:08.129810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:08.129640    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 18:52:09.123857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:09.123823    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:52:09.125081 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:09.125058    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:52:22.029787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:22.029752    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"]
Oct 30 18:52:22.034994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:22.034967    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae71fd2-lpvh9"]
Oct 30 18:52:23.234695 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:52:23.234658    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:52:23.895961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:23.895903    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73096e9-d6ec-4b22-af51-00d1da4110db" path="/var/lib/kubelet/pods/f73096e9-d6ec-4b22-af51-00d1da4110db/volumes"
Oct 30 18:52:33.048212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:33.048170    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"]
Oct 30 18:52:33.051542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:33.051507    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca49098-9rwf5"]
Oct 30 18:52:33.967804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:33.967774    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260a295b-a834-4498-9846-db028879c221" path="/var/lib/kubelet/pods/260a295b-a834-4498-9846-db028879c221/volumes"
Oct 30 18:52:35.249190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.249145    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"]
Oct 30 18:52:35.274924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.274885    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d440be8-rx6kn"]
Oct 30 18:52:35.332356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.331731    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:52:35.751112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.750724    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:35.762756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.754069    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 30 18:52:35.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.827611    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:35.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.827712    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njh5g\" (UniqueName: \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:35.875114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.870218    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8fd4ed\""
Oct 30 18:52:35.931185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.930978    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:35.931185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.931065    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njh5g\" (UniqueName: \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:35.969492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:35.932775    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:36.042984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.042950    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njh5g\" (UniqueName: \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:36.102485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.102453    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa761428-0a3c-41a3-8ea0-6a084ad3d07f" path="/var/lib/kubelet/pods/fa761428-0a3c-41a3-8ea0-6a084ad3d07f/volumes"
Oct 30 18:52:36.118119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.111006    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:52:36.118489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.118468    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:52:36.121043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.121022    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:36.173546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.173509    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.197789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.197763    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8fd4ed-dockercfg-fb652\""
Oct 30 18:52:36.198017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.197999    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:52:36.274185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.274154    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjs9\" (UniqueName: \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.274630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.274217    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.377441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.376577    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjs9\" (UniqueName: \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.377441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.376649    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.378242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.378179    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.592166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.573797    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjs9\" (UniqueName: \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\") pod \"payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:36.895366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:36.894319    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:37.259349 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.259318    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:52:37.336755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.336724    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" event={"ID":"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb","Type":"ContainerStarted","Data":"4614915f0c3a757f2b2f21fe2e40398f7098c83a94c7cccc58c8904f00717ebe"}
Oct 30 18:52:37.442995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.442953    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:52:37.443282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.443251    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy" containerID="cri-o://592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636" gracePeriod=30
Oct 30 18:52:37.452123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.451195    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:52:37.452123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.451423    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy" containerID="cri-o://e9f506cb7dcce5ed9ab12a9174661c8215dea0a8fec7d46e5ba73c287c07a053" gracePeriod=30
Oct 30 18:52:37.632766 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:52:37.632735    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b6aeb7_8c81_46a0_82b6_252f64d39492.slice/crio-6998a11869e800f6bd4f7444ed8028206ae2cd0e651d37a10ea68eea5a52309a WatchSource:0}: Error finding container 6998a11869e800f6bd4f7444ed8028206ae2cd0e651d37a10ea68eea5a52309a: Status 404 returned error can't find the container with id 6998a11869e800f6bd4f7444ed8028206ae2cd0e651d37a10ea68eea5a52309a
Oct 30 18:52:37.634690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:37.634637    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:52:38.349186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.349148    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" event={"ID":"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb","Type":"ContainerStarted","Data":"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"}
Oct 30 18:52:38.349656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.349310    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:38.351420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.351398    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" event={"ID":"e2b6aeb7-8c81-46a0-82b6-252f64d39492","Type":"ContainerStarted","Data":"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"}
Oct 30 18:52:38.351559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.351435    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" event={"ID":"e2b6aeb7-8c81-46a0-82b6-252f64d39492","Type":"ContainerStarted","Data":"6998a11869e800f6bd4f7444ed8028206ae2cd0e651d37a10ea68eea5a52309a"}
Oct 30 18:52:38.351912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.351897    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:38.377128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.377053    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" podStartSLOduration=3.377037675 podStartE2EDuration="3.377037675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:38.372399941 +0000 UTC m=+1178.635864889" watchObservedRunningTime="2026-10-30 18:52:38.377037675 +0000 UTC m=+1178.640502623"
Oct 30 18:52:38.732237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.731073    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" podStartSLOduration=3.731053267 podStartE2EDuration="3.731053267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:38.399814669 +0000 UTC m=+1178.663279623" watchObservedRunningTime="2026-10-30 18:52:38.731053267 +0000 UTC m=+1178.994518216"
Oct 30 18:52:38.759628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.754464    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"]
Oct 30 18:52:38.763383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.763360    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.770867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.770843    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"]
Oct 30 18:52:38.771422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.771399    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:52:38.771523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.771465    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:38.791440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.787959    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.791440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.788026    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvg9\" (UniqueName: \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.791440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.788157    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.901079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.901042    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.901371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.901347    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvg9\" (UniqueName: \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.901584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.901569    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.904520 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.904494    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.908321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.908274    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:38.928119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:38.928049    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvg9\" (UniqueName: \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\") pod \"maas-api-revoke-sub-7ccc85d1-t6kgz\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:39.128688 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:39.128379    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:39.573140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:39.573104    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"]
Oct 30 18:52:39.576677 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:52:39.576644    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd509ea9f_1c78_41da_a66e_5013cda08ad5.slice/crio-87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4 WatchSource:0}: Error finding container 87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4: Status 404 returned error can't find the container with id 87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4
Oct 30 18:52:40.385145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:40.379225    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz" event={"ID":"d509ea9f-1c78-41da-a66e-5013cda08ad5","Type":"ContainerStarted","Data":"d579244d4ad94655c47e7c13d48e6df678a9ab8d493af42d73436e6006fbda8e"}
Oct 30 18:52:40.385145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:40.379289    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz" event={"ID":"d509ea9f-1c78-41da-a66e-5013cda08ad5","Type":"ContainerStarted","Data":"87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4"}
Oct 30 18:52:40.418828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:40.415629    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz" podStartSLOduration=2.4156002340000002 podStartE2EDuration="2.415600234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.401640162 +0000 UTC m=+1180.665105127" watchObservedRunningTime="2026-10-30 18:52:40.415600234 +0000 UTC m=+1180.679065181"
Oct 30 18:52:41.408060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:41.407970    5065 generic.go:356] "Generic (PLEG): container finished" podID="d509ea9f-1c78-41da-a66e-5013cda08ad5" containerID="d579244d4ad94655c47e7c13d48e6df678a9ab8d493af42d73436e6006fbda8e" exitCode=0
Oct 30 18:52:41.410363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:41.408225    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz" event={"ID":"d509ea9f-1c78-41da-a66e-5013cda08ad5","Type":"ContainerDied","Data":"d579244d4ad94655c47e7c13d48e6df678a9ab8d493af42d73436e6006fbda8e"}
Oct 30 18:52:42.028900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.028673    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"]
Oct 30 18:52:42.037598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.037573    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e1bae1-xlvdr"]
Oct 30 18:52:42.725782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.725347    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:42.856966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.856936    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\") pod \"d509ea9f-1c78-41da-a66e-5013cda08ad5\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") "
Oct 30 18:52:42.857302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.857274    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca" pod "d509ea9f-1c78-41da-a66e-5013cda08ad5" (UID: "d509ea9f-1c78-41da-a66e-5013cda08ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:42.857466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.857287    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\" (UniqueName: \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\") pod \"d509ea9f-1c78-41da-a66e-5013cda08ad5\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") "
Oct 30 18:52:42.857747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.857726    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\") pod \"d509ea9f-1c78-41da-a66e-5013cda08ad5\" (UID: \"d509ea9f-1c78-41da-a66e-5013cda08ad5\") "
Oct 30 18:52:42.859745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.859629    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:52:42.860176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.860134    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:42.861416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.861378    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca" pod "d509ea9f-1c78-41da-a66e-5013cda08ad5" (UID: "d509ea9f-1c78-41da-a66e-5013cda08ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:42.862948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.862921    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9" pod "d509ea9f-1c78-41da-a66e-5013cda08ad5" (UID: "d509ea9f-1c78-41da-a66e-5013cda08ad5"). InnerVolumeSpecName "kube-api-access-6bvg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:42.961454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961372    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961432    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961545    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961683    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961717    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961755    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961788    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.961879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.961859    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\") pod \"41c896eb-26e8-4242-b81c-52336f28e3a8\" (UID: \"41c896eb-26e8-4242-b81c-52336f28e3a8\") "
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962441    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962516    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvg9\" (UniqueName: \"kubernetes.io/projected/d509ea9f-1c78-41da-a66e-5013cda08ad5-kube-api-access-6bvg9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962537    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d509ea9f-1c78-41da-a66e-5013cda08ad5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962573    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962585    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962712    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:42.962755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.962727    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:42.965624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.965584    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:42.965784 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.965761    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:52:42.965862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.965797    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "kube-api-access-f8m4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:42.966068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:42.966043    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy" pod "41c896eb-26e8-4242-b81c-52336f28e3a8" (UID: "41c896eb-26e8-4242-b81c-52336f28e3a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064393    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064428    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064445    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064460    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c896eb-26e8-4242-b81c-52336f28e3a8-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064473    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064486    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064499    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8m4w\" (UniqueName: \"kubernetes.io/projected/41c896eb-26e8-4242-b81c-52336f28e3a8-kube-api-access-f8m4w\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064513    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c896eb-26e8-4242-b81c-52336f28e3a8-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.064553 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.064526    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c896eb-26e8-4242-b81c-52336f28e3a8-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.441600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.441326    5065 generic.go:356] "Generic (PLEG): container finished" podID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerID="e9f506cb7dcce5ed9ab12a9174661c8215dea0a8fec7d46e5ba73c287c07a053" exitCode=0
Oct 30 18:52:43.441600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.441416    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerDied","Data":"e9f506cb7dcce5ed9ab12a9174661c8215dea0a8fec7d46e5ba73c287c07a053"}
Oct 30 18:52:43.441600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.441483    5065 scope.go:122] "RemoveContainer" containerID="4d01a3153dd6e2f2e2e64c02dbc6bbbaaca0a1c770e970780841dd279bf93494"
Oct 30 18:52:43.452138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.449579    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz" event={"ID":"d509ea9f-1c78-41da-a66e-5013cda08ad5","Type":"ContainerDied","Data":"87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4"}
Oct 30 18:52:43.452138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.449641    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f23e31f9dd0c493a1ff14d5d3751cb7ca48d9e24f2fd98a6b42ff8916bd7a4"
Oct 30 18:52:43.452138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.449721    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"
Oct 30 18:52:43.457708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.454307    5065 generic.go:356] "Generic (PLEG): container finished" podID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerID="592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636" exitCode=0
Oct 30 18:52:43.457708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.454534    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" event={"ID":"41c896eb-26e8-4242-b81c-52336f28e3a8","Type":"ContainerDied","Data":"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"}
Oct 30 18:52:43.457708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.454574    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw" event={"ID":"41c896eb-26e8-4242-b81c-52336f28e3a8","Type":"ContainerDied","Data":"240bd0b38f22e35372af8775bf1c7ead9493c35d771cac2bca1b78aa79ac4aa7"}
Oct 30 18:52:43.457708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.454649    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"
Oct 30 18:52:43.493206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.492990    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:52:43.527538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.527513    5065 scope.go:122] "RemoveContainer" containerID="592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"
Oct 30 18:52:43.547947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.547923    5065 scope.go:122] "RemoveContainer" containerID="592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"
Oct 30 18:52:43.548360 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:52:43.548327    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636\": container with ID starting with 592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636 not found: ID does not exist" containerID="592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"
Oct 30 18:52:43.548484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.548370    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636"} err="failed to get container status \"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636\": rpc error: code = NotFound desc = could not find container \"592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636\": container with ID starting with 592379ddb24ea5d246d5f0f123cd36f410d2d03aa39778e4c9466d0f8af39636 not found: ID does not exist"
Oct 30 18:52:43.548559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.548481    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:52:43.565470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.565437    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wd8zw"]
Oct 30 18:52:43.615615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.615580    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.617328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.617305    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.617973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.617953    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.618750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.618711    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.619340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.619323    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.619975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.616108    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.620623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.619231    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:43.620749 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.620272    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.620985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.620937    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:52:43.621165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.621137    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.621398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.621344    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.621929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.621909    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.622533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.622511    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\") pod \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\" (UID: \"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95\") "
Oct 30 18:52:43.623733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.621825    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.624164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.622317    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.624377 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.624061    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.624547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.624532    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.624703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.624688    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.624855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.624820    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.635143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.629455    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:43.635461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.635390    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "kube-api-access-5c77l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:43.636149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.636113    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy" pod "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" (UID: "bf9c71a8-8cf5-44d2-b254-3d27b08b7f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:43.726796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.726760    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.726796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.726790    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c77l\" (UniqueName: \"kubernetes.io/projected/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-kube-api-access-5c77l\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.726796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.726800    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.726796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.726808    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.726796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.726819    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:43.895765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.895724    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" path="/var/lib/kubelet/pods/41c896eb-26e8-4242-b81c-52336f28e3a8/volumes"
Oct 30 18:52:43.896489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:43.896463    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b788440-3bf0-4f25-9895-4e60e92c8633" path="/var/lib/kubelet/pods/5b788440-3bf0-4f25-9895-4e60e92c8633/volumes"
Oct 30 18:52:44.467373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:44.467288    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"
Oct 30 18:52:44.467373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:44.467319    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s" event={"ID":"bf9c71a8-8cf5-44d2-b254-3d27b08b7f95","Type":"ContainerDied","Data":"84342a9c4ea28e4b29ea2dc69d561a27979b189cea02ffbc74b46844b197e84f"}
Oct 30 18:52:44.492063 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:44.467378    5065 scope.go:122] "RemoveContainer" containerID="e9f506cb7dcce5ed9ab12a9174661c8215dea0a8fec7d46e5ba73c287c07a053"
Oct 30 18:52:44.525950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:44.525922    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:52:44.531396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:44.531374    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mz99s"]
Oct 30 18:52:45.478278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.478249    5065 generic.go:356] "Generic (PLEG): container finished" podID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerID="1a374073e324e792b690251fe0a626cad3a6c0d4491a923cb2a2ad90ff85a53b" exitCode=0
Oct 30 18:52:45.478720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.478323    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerDied","Data":"1a374073e324e792b690251fe0a626cad3a6c0d4491a923cb2a2ad90ff85a53b"}
Oct 30 18:52:45.478778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.478734    5065 scope.go:122] "RemoveContainer" containerID="1a374073e324e792b690251fe0a626cad3a6c0d4491a923cb2a2ad90ff85a53b"
Oct 30 18:52:45.480328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.480268    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.480449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.480363    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.480449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.480428    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.481637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.481614    5065 generic.go:356] "Generic (PLEG): container finished" podID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerID="8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693" exitCode=0
Oct 30 18:52:45.481723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.481686    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerDied","Data":"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"}
Oct 30 18:52:45.482071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.482055    5065 scope.go:122] "RemoveContainer" containerID="8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"
Oct 30 18:52:45.484260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.484230    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.484339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.484310    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.484382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.484353    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:52:45.901111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:45.897337    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" path="/var/lib/kubelet/pods/bf9c71a8-8cf5-44d2-b254-3d27b08b7f95/volumes"
Oct 30 18:52:46.257282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.257194    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:46.468598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.468565    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:52:46.469405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469385    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.469405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469404    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.469582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469434    5065 cpu_manager.go:414] "containerMap: removing container" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy"
Oct 30 18:52:46.469582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469442    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy"
Oct 30 18:52:46.469582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469455    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d509ea9f-1c78-41da-a66e-5013cda08ad5" containerName="revoke-keys"
Oct 30 18:52:46.469582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469462    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d509ea9f-1c78-41da-a66e-5013cda08ad5" containerName="revoke-keys"
Oct 30 18:52:46.469790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469696    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.469790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469715    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c896eb-26e8-4242-b81c-52336f28e3a8" containerName="istio-proxy"
Oct 30 18:52:46.469790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.469727    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d509ea9f-1c78-41da-a66e-5013cda08ad5" containerName="revoke-keys"
Oct 30 18:52:46.473568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.473546    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.555116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.549425    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ceefc\""
Oct 30 18:52:46.555116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.550298    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ceefc-dockercfg-bbcl4\""
Oct 30 18:52:46.613396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.577590    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.613396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.577950    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6v2r\" (UniqueName: \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.636769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.636732    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:52:46.813479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.711001    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.813479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.711197    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6v2r\" (UniqueName: \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.813479 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.712544    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.878078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.878015    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerStarted","Data":"23125a2854f1b4f3a441bf701aa74c468039e98b8b0a5b841377bcb065b91d53"}
Oct 30 18:52:46.879000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.878981    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:52:46.931140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.919446    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.931140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.919480    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.931140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.919789    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9c71a8-8cf5-44d2-b254-3d27b08b7f95" containerName="istio-proxy"
Oct 30 18:52:46.945615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.943399    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6v2r\" (UniqueName: \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\") pod \"payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:46.962357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.962331    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:52:46.962777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.962497    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:46.965019 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:46.964991    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerStarted","Data":"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"}
Oct 30 18:52:47.043833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.043799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.044413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.044325    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpgf\" (UniqueName: \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.082566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.082488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:47.140907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.140856    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.141295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.141220    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:47.152806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.149719    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpgf\" (UniqueName: \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.153588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.153550    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.154909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.154865    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.174046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.174015    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:47.209744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.180646    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpgf\" (UniqueName: \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.319665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.319365    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:47.628530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.618858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:52:47.749867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.749842    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:52:47.852134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.851450    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:47.852459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.852432    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.852557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:47.852484    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 18:52:48.066573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.066474    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" event={"ID":"ab0a2ebd-912c-4c20-89af-cd10830e7118","Type":"ContainerStarted","Data":"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"}
Oct 30 18:52:48.066573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.066539    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" event={"ID":"ab0a2ebd-912c-4c20-89af-cd10830e7118","Type":"ContainerStarted","Data":"b6253466c4b9ae788209f6f990f92ae32b289e57068a7279f6c89817446b3043"}
Oct 30 18:52:48.069186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.069162    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:52:48.071627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.071605    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:48.071769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.071654    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" event={"ID":"12ee5981-c096-4bf6-9418-7a5d5c0e1100","Type":"ContainerStarted","Data":"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"}
Oct 30 18:52:48.071769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.071693    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" event={"ID":"12ee5981-c096-4bf6-9418-7a5d5c0e1100","Type":"ContainerStarted","Data":"92bef74483b283b147886b9c501566f07d7684296ad749099136a1353e343aa2"}
Oct 30 18:52:48.074414 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.074393    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:48.074570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.074551    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:48.196679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.196566    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" podStartSLOduration=2.196546756 podStartE2EDuration="2.196546756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:48.174407961 +0000 UTC m=+1188.437872935" watchObservedRunningTime="2026-10-30 18:52:48.196546756 +0000 UTC m=+1188.460011706"
Oct 30 18:52:48.202277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.202227    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" podStartSLOduration=2.202216019 podStartE2EDuration="2.202216019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:48.14019697 +0000 UTC m=+1188.403661939" watchObservedRunningTime="2026-10-30 18:52:48.202216019 +0000 UTC m=+1188.465680969"
Oct 30 18:52:48.857110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.856618    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 18:52:48.857110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:48.856673    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 18:52:49.085492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.085456    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:49.085780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.085747    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:49.345133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.344771    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:49.369448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.369417    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:52:49.851624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.851575    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 18:52:49.851843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:49.851636    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 18:52:50.074350 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:50.074316    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:50.074741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:50.074382    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:50.397989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:50.394223    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:52:50.851097 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:50.851063    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 18:52:50.851295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:50.851134    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 18:52:51.075012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:51.074979    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:51.075503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:51.075038    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:51.850934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:51.850891    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 18:52:51.851128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:51.850966    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 18:52:52.077438 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:52.077399    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:52:52.078043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:52.078012    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 18:52:52.855871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:52.855844    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:52.856872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:52.856853    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:52:53.059258 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:52:53.059222    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:52:53.079538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:53.079507    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:53.079986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:53.079977    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:53.080782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:53.080764    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:52:54.032607 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:54.032569    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"]
Oct 30 18:52:54.037736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:54.037704    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772b3bcb-n82n2"]
Oct 30 18:52:55.109943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:55.109905    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"]
Oct 30 18:52:55.138782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:55.132508    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d2ece6-bvwl6"]
Oct 30 18:52:55.893395 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:55.893358    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd78a9a-d401-4a80-8b0b-cde6de268bd1" path="/var/lib/kubelet/pods/5fd78a9a-d401-4a80-8b0b-cde6de268bd1/volumes"
Oct 30 18:52:55.893936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:55.893911    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68e66d7-4067-470a-bfd0-465009e59612" path="/var/lib/kubelet/pods/f68e66d7-4067-470a-bfd0-465009e59612/volumes"
Oct 30 18:52:56.056053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.056023    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"]
Oct 30 18:52:56.061968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.061946    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.069247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.069220    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:56.069247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.069227    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:52:56.077860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.077840    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"]
Oct 30 18:52:56.130102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.130050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.130492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.130296    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2z5\" (UniqueName: \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.130492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.130391    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.231846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.231765    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2z5\" (UniqueName: \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.232007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.231854    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.232007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.231926    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.232692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.232669    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.234397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.234374    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.245383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.245360    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2z5\" (UniqueName: \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.378654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.378625    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:56.674157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:56.674131    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"]
Oct 30 18:52:56.675184 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:52:56.675156    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde56f31e_f535_4aa3_b14b_65233639d6ed.slice/crio-561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804 WatchSource:0}: Error finding container 561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804: Status 404 returned error can't find the container with id 561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804
Oct 30 18:52:57.338837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.338801    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt" event={"ID":"de56f31e-f535-4aa3-b14b-65233639d6ed","Type":"ContainerStarted","Data":"325d50b7c62d85e4f66c8fc62b26690393a93ca9e6f417620742f5401c846e3f"}
Oct 30 18:52:57.339262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.338870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt" event={"ID":"de56f31e-f535-4aa3-b14b-65233639d6ed","Type":"ContainerStarted","Data":"561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804"}
Oct 30 18:52:57.361364 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.361317    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt" podStartSLOduration=1.361305773 podStartE2EDuration="1.361305773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:57.360396398 +0000 UTC m=+1197.623861365" watchObservedRunningTime="2026-10-30 18:52:57.361305773 +0000 UTC m=+1197.624770720"
Oct 30 18:52:57.809297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.809263    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"]
Oct 30 18:52:57.813957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.813934    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.826698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.826675    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"]
Oct 30 18:52:57.852585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.852555    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlwm\" (UniqueName: \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.852585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.852589    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.852779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.852759    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.954159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.954062    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.954459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.954385    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlwm\" (UniqueName: \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.954459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.954427    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.955009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.954988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.956708 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.956683    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:57.968354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:57.968331    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlwm\" (UniqueName: \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:58.134313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:58.134278    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:52:58.357802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:58.357765    5065 generic.go:356] "Generic (PLEG): container finished" podID="de56f31e-f535-4aa3-b14b-65233639d6ed" containerID="325d50b7c62d85e4f66c8fc62b26690393a93ca9e6f417620742f5401c846e3f" exitCode=0
Oct 30 18:52:58.358364 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:58.358279    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt" event={"ID":"de56f31e-f535-4aa3-b14b-65233639d6ed","Type":"ContainerDied","Data":"325d50b7c62d85e4f66c8fc62b26690393a93ca9e6f417620742f5401c846e3f"}
Oct 30 18:52:58.533507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:58.533476    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"]
Oct 30 18:52:59.037905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.037875    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"]
Oct 30 18:52:59.042844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.042603    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e39a61ed-87qct"]
Oct 30 18:52:59.136344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.136318    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:52:59.372044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.371961    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn" event={"ID":"89423b5c-8a43-40cf-ada8-c16c1a54aaae","Type":"ContainerStarted","Data":"ff2693da27b3edafe31a755b7dc04d9942188cd6825c5c3921555b8ae94caded"}
Oct 30 18:52:59.372044 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.372019    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn" event={"ID":"89423b5c-8a43-40cf-ada8-c16c1a54aaae","Type":"ContainerStarted","Data":"f813a2539f08fd34bc2f32dcb2bbf061a20a322b935d10d38cc8f5f93a5c4897"}
Oct 30 18:52:59.405034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.404896    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn" podStartSLOduration=2.404877526 podStartE2EDuration="2.404877526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:59.399910638 +0000 UTC m=+1199.663375654" watchObservedRunningTime="2026-10-30 18:52:59.404877526 +0000 UTC m=+1199.668342476"
Oct 30 18:52:59.773996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.773644    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:52:59.829775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.829733    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\") pod \"de56f31e-f535-4aa3-b14b-65233639d6ed\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") "
Oct 30 18:52:59.829983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.829860    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\" (UniqueName: \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\") pod \"de56f31e-f535-4aa3-b14b-65233639d6ed\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") "
Oct 30 18:52:59.830137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.830116    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\") pod \"de56f31e-f535-4aa3-b14b-65233639d6ed\" (UID: \"de56f31e-f535-4aa3-b14b-65233639d6ed\") "
Oct 30 18:52:59.831035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.830994    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca" pod "de56f31e-f535-4aa3-b14b-65233639d6ed" (UID: "de56f31e-f535-4aa3-b14b-65233639d6ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:59.834346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.834315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5" pod "de56f31e-f535-4aa3-b14b-65233639d6ed" (UID: "de56f31e-f535-4aa3-b14b-65233639d6ed"). InnerVolumeSpecName "kube-api-access-wx2z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:59.836172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.836146    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca" pod "de56f31e-f535-4aa3-b14b-65233639d6ed" (UID: "de56f31e-f535-4aa3-b14b-65233639d6ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:59.836353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.836330    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:59.836432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.836358    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2z5\" (UniqueName: \"kubernetes.io/projected/de56f31e-f535-4aa3-b14b-65233639d6ed-kube-api-access-wx2z5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:59.836432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.836374    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de56f31e-f535-4aa3-b14b-65233639d6ed-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:59.903112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:52:59.899198    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0980cfb-4d85-4d54-b7a3-5f149d81956f" path="/var/lib/kubelet/pods/f0980cfb-4d85-4d54-b7a3-5f149d81956f/volumes"
Oct 30 18:53:00.158115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.157292    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:53:00.386187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.386149    5065 generic.go:356] "Generic (PLEG): container finished" podID="89423b5c-8a43-40cf-ada8-c16c1a54aaae" containerID="ff2693da27b3edafe31a755b7dc04d9942188cd6825c5c3921555b8ae94caded" exitCode=0
Oct 30 18:53:00.386699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.386254    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn" event={"ID":"89423b5c-8a43-40cf-ada8-c16c1a54aaae","Type":"ContainerDied","Data":"ff2693da27b3edafe31a755b7dc04d9942188cd6825c5c3921555b8ae94caded"}
Oct 30 18:53:00.388823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.388752    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt" event={"ID":"de56f31e-f535-4aa3-b14b-65233639d6ed","Type":"ContainerDied","Data":"561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804"}
Oct 30 18:53:00.388952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.388827    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561260976cdcd585e68797c7a25d7fb521cd1cb5779f8fbd03953ff194d7f804"
Oct 30 18:53:00.389034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:00.388982    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"
Oct 30 18:53:01.004111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.000036    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"]
Oct 30 18:53:01.004111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.001303    5065 cpu_manager.go:414] "containerMap: removing container" podUID="de56f31e-f535-4aa3-b14b-65233639d6ed" containerName="revoke-keys"
Oct 30 18:53:01.004111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.001323    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de56f31e-f535-4aa3-b14b-65233639d6ed" containerName="revoke-keys"
Oct 30 18:53:01.004111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.001525    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="de56f31e-f535-4aa3-b14b-65233639d6ed" containerName="revoke-keys"
Oct 30 18:53:01.010128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.009647    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.023751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.022074    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"]
Oct 30 18:53:01.069854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.069818    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.070133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.070056    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.070133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.070122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2p8\" (UniqueName: \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.171836 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.171801    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.172009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.171856    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.172009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.171884    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2p8\" (UniqueName: \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.172710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.172565    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.174590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.174567    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.248022 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.247818    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2p8\" (UniqueName: \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:01.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:01.392777    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:02.113148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.112758    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:53:02.161505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.161472    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\") pod \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") "
Oct 30 18:53:02.161679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.161531    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\" (UniqueName: \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\") pod \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") "
Oct 30 18:53:02.161679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.161579    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\") pod \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\" (UID: \"89423b5c-8a43-40cf-ada8-c16c1a54aaae\") "
Oct 30 18:53:02.162358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.162317    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca" pod "89423b5c-8a43-40cf-ada8-c16c1a54aaae" (UID: "89423b5c-8a43-40cf-ada8-c16c1a54aaae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:02.164182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.164154    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca" pod "89423b5c-8a43-40cf-ada8-c16c1a54aaae" (UID: "89423b5c-8a43-40cf-ada8-c16c1a54aaae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:02.164293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.164176    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm" pod "89423b5c-8a43-40cf-ada8-c16c1a54aaae" (UID: "89423b5c-8a43-40cf-ada8-c16c1a54aaae"). InnerVolumeSpecName "kube-api-access-4nlwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:02.267043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.266972    5065 scope.go:122] "RemoveContainer" containerID="3ca0372ae2dd62d33914b058887d819b06f9603ac3b3e241466edeee4ebf09f7"
Oct 30 18:53:02.268206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.267519    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:02.268206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.267542    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nlwm\" (UniqueName: \"kubernetes.io/projected/89423b5c-8a43-40cf-ada8-c16c1a54aaae-kube-api-access-4nlwm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:02.268206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.267552    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89423b5c-8a43-40cf-ada8-c16c1a54aaae-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:02.365293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.362336    5065 scope.go:122] "RemoveContainer" containerID="924bcf52fcb4b2c82473a695d7f43b98a088e0878dc43794b85c1ea3a552cf28"
Oct 30 18:53:02.366296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.366261    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"]
Oct 30 18:53:02.402975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.402407    5065 scope.go:122] "RemoveContainer" containerID="6194fadb229c0d869a6344102383e87fe04053703d03dc8ace69e53446ec8f2f"
Oct 30 18:53:02.478916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.478355    5065 scope.go:122] "RemoveContainer" containerID="751e21d5d8060b000246bb4435c86e29311f64a91ff5d7889ad5739bb8c6a8de"
Oct 30 18:53:02.499228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.498801    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv" event={"ID":"b44cdafd-adca-4c9b-a880-a1266843d54d","Type":"ContainerStarted","Data":"970229ded390f70e2882eba3e74fd443aa0f22ac93b4cd82f9249b5731583261"}
Oct 30 18:53:02.502297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.502263    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn" event={"ID":"89423b5c-8a43-40cf-ada8-c16c1a54aaae","Type":"ContainerDied","Data":"f813a2539f08fd34bc2f32dcb2bbf061a20a322b935d10d38cc8f5f93a5c4897"}
Oct 30 18:53:02.502428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.502320    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f813a2539f08fd34bc2f32dcb2bbf061a20a322b935d10d38cc8f5f93a5c4897"
Oct 30 18:53:02.502428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:02.502393    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"
Oct 30 18:53:03.264564 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.264535    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:53:03.273971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.273940    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:53:03.396399 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.396074    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:53:03.406665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.406636    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:53:03.520694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.520667    5065 generic.go:356] "Generic (PLEG): container finished" podID="b44cdafd-adca-4c9b-a880-a1266843d54d" containerID="1f18fa0b4e9fbdfaca1a3746ca5861d6cca5733c85371cbc122be302c5f045ae" exitCode=0
Oct 30 18:53:03.520832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:03.520751    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv" event={"ID":"b44cdafd-adca-4c9b-a880-a1266843d54d","Type":"ContainerDied","Data":"1f18fa0b4e9fbdfaca1a3746ca5861d6cca5733c85371cbc122be302c5f045ae"}
Oct 30 18:53:04.088557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.088102    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"]
Oct 30 18:53:04.097668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.097634    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76db0ccd-c9v62"]
Oct 30 18:53:04.706727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.706691    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:04.873157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.873059    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\") pod \"b44cdafd-adca-4c9b-a880-a1266843d54d\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") "
Oct 30 18:53:04.873307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.873202    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\" (UniqueName: \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\") pod \"b44cdafd-adca-4c9b-a880-a1266843d54d\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") "
Oct 30 18:53:04.873359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.873314    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\") pod \"b44cdafd-adca-4c9b-a880-a1266843d54d\" (UID: \"b44cdafd-adca-4c9b-a880-a1266843d54d\") "
Oct 30 18:53:04.873661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.873638    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca" pod "b44cdafd-adca-4c9b-a880-a1266843d54d" (UID: "b44cdafd-adca-4c9b-a880-a1266843d54d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:04.873847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.873830    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:04.875331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.875307    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca" pod "b44cdafd-adca-4c9b-a880-a1266843d54d" (UID: "b44cdafd-adca-4c9b-a880-a1266843d54d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:04.875494 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.875478    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8" pod "b44cdafd-adca-4c9b-a880-a1266843d54d" (UID: "b44cdafd-adca-4c9b-a880-a1266843d54d"). InnerVolumeSpecName "kube-api-access-gw2p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:04.993166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.991203    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44cdafd-adca-4c9b-a880-a1266843d54d-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:04.993166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:04.991239    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2p8\" (UniqueName: \"kubernetes.io/projected/b44cdafd-adca-4c9b-a880-a1266843d54d-kube-api-access-gw2p8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:05.539441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:05.539414    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"
Oct 30 18:53:05.539441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:05.539428    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv" event={"ID":"b44cdafd-adca-4c9b-a880-a1266843d54d","Type":"ContainerDied","Data":"970229ded390f70e2882eba3e74fd443aa0f22ac93b4cd82f9249b5731583261"}
Oct 30 18:53:05.539659 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:05.539512    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="970229ded390f70e2882eba3e74fd443aa0f22ac93b4cd82f9249b5731583261"
Oct 30 18:53:05.896179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:05.896070    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870001e6-feb9-4909-961b-953d412bc413" path="/var/lib/kubelet/pods/870001e6-feb9-4909-961b-953d412bc413/volumes"
Oct 30 18:53:06.044030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:06.043988    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"]
Oct 30 18:53:06.046660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:06.046638    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4672b909-qvms8"]
Oct 30 18:53:07.895532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:07.895497    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb885de6-dcde-4d90-a0e9-364e37f80aa7" path="/var/lib/kubelet/pods/cb885de6-dcde-4d90-a0e9-364e37f80aa7/volumes"
Oct 30 18:53:08.856359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.856330    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:08.857030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857015    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b44cdafd-adca-4c9b-a880-a1266843d54d" containerName="revoke-keys"
Oct 30 18:53:08.857068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857032    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44cdafd-adca-4c9b-a880-a1266843d54d" containerName="revoke-keys"
Oct 30 18:53:08.857167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857068    5065 cpu_manager.go:414] "containerMap: removing container" podUID="89423b5c-8a43-40cf-ada8-c16c1a54aaae" containerName="revoke-keys"
Oct 30 18:53:08.857167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857073    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89423b5c-8a43-40cf-ada8-c16c1a54aaae" containerName="revoke-keys"
Oct 30 18:53:08.857236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857207    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="89423b5c-8a43-40cf-ada8-c16c1a54aaae" containerName="revoke-keys"
Oct 30 18:53:08.857236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.857222    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44cdafd-adca-4c9b-a880-a1266843d54d" containerName="revoke-keys"
Oct 30 18:53:08.862352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.862332    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:08.873719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.873696    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:08.935959 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.935936    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-348877-dockercfg-grl5w\""
Oct 30 18:53:08.956129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.955606    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qkk\" (UniqueName: \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:08.956129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.955751    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:08.976124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:08.975103    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-348877\""
Oct 30 18:53:09.062110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.058076    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:09.062110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.058440    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qkk\" (UniqueName: \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:09.062110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.060051    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:09.114179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.113820    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qkk\" (UniqueName: \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\") pod \"payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:09.187259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.185672    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:09.510018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.509990    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:09.576316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:09.576286    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" event={"ID":"8d624ad4-d108-4ec6-a60b-9e112960bb3f","Type":"ContainerStarted","Data":"ce4d425dfc51a04799ab16b7776a98ea5d3259ecb22df1a8824b52e24657679f"}
Oct 30 18:53:10.028123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.028075    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"]
Oct 30 18:53:10.032615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.032592    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cf63931-cp4fr"]
Oct 30 18:53:10.487096 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.486978    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:53:10.487247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.487184    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" podUID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" containerName="payload-processing" containerID="cri-o://3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76" gracePeriod=30
Oct 30 18:53:10.500275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.500251    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:53:10.500455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.500436    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" containerName="payload-pre-processing" containerID="cri-o://129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e" gracePeriod=30
Oct 30 18:53:10.588009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.587976    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" event={"ID":"8d624ad4-d108-4ec6-a60b-9e112960bb3f","Type":"ContainerStarted","Data":"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"}
Oct 30 18:53:10.588227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.588208    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:10.624743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.624690    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" podStartSLOduration=2.624676714 podStartE2EDuration="2.624676714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:10.617659473 +0000 UTC m=+1210.881124423" watchObservedRunningTime="2026-10-30 18:53:10.624676714 +0000 UTC m=+1210.888141698"
Oct 30 18:53:10.791177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.788356    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.160:9004: connect: connection refused"
Oct 30 18:53:10.964118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.964051    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"]
Oct 30 18:53:10.979766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.977968    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52fa8b-vc6vt"]
Oct 30 18:53:11.005130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:10.988585    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:53:11.081806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.081690    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\") pod \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") "
Oct 30 18:53:11.082745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.082705    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\" (UniqueName: \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\") pod \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\" (UID: \"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c\") "
Oct 30 18:53:11.084663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.084619    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume" pod "0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" (UID: "0e0767d5-b349-45cf-9f6c-47ad7bcadd9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:11.124487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.124456    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"]
Oct 30 18:53:11.135787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.132236    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"]
Oct 30 18:53:11.141702 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.141677    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5b5294-5cqmh"]
Oct 30 18:53:11.145008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.144987    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b3ea6f-bjch7"]
Oct 30 18:53:11.161295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.161257    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9" pod "0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" (UID: "0e0767d5-b349-45cf-9f6c-47ad7bcadd9c"). InnerVolumeSpecName "kube-api-access-vskp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:11.186343 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.186274    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vskp9\" (UniqueName: \"kubernetes.io/projected/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-kube-api-access-vskp9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:11.186343 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.186314    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:11.209937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.209914    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:53:11.286977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.286943    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\" (UniqueName: \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\") pod \"b5df24f1-8904-4d57-9874-bea45240fa86\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") "
Oct 30 18:53:11.287204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.287179    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\") pod \"b5df24f1-8904-4d57-9874-bea45240fa86\" (UID: \"b5df24f1-8904-4d57-9874-bea45240fa86\") "
Oct 30 18:53:11.287621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.287592    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume" pod "b5df24f1-8904-4d57-9874-bea45240fa86" (UID: "b5df24f1-8904-4d57-9874-bea45240fa86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:11.287858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.287839    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df24f1-8904-4d57-9874-bea45240fa86-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:11.290059 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.290027    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf" pod "b5df24f1-8904-4d57-9874-bea45240fa86" (UID: "b5df24f1-8904-4d57-9874-bea45240fa86"). InnerVolumeSpecName "kube-api-access-dm5xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:11.389493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.389426    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5xf\" (UniqueName: \"kubernetes.io/projected/b5df24f1-8904-4d57-9874-bea45240fa86-kube-api-access-dm5xf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:11.614819 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.614784    5065 generic.go:356] "Generic (PLEG): container finished" podID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" containerID="3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76" exitCode=0
Oct 30 18:53:11.615010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.614853    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"
Oct 30 18:53:11.615010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.614871    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" event={"ID":"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c","Type":"ContainerDied","Data":"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"}
Oct 30 18:53:11.615010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.614964    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc" event={"ID":"0e0767d5-b349-45cf-9f6c-47ad7bcadd9c","Type":"ContainerDied","Data":"40a64ad83eca835abc3dc14f6bd83c4d476f4dc7a849d3205d9c91fb2f3e499d"}
Oct 30 18:53:11.615010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.614992    5065 scope.go:122] "RemoveContainer" containerID="3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"
Oct 30 18:53:11.616562 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.616542    5065 generic.go:356] "Generic (PLEG): container finished" podID="b5df24f1-8904-4d57-9874-bea45240fa86" containerID="129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e" exitCode=0
Oct 30 18:53:11.616678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.616601    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"
Oct 30 18:53:11.616678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.616635    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" event={"ID":"b5df24f1-8904-4d57-9874-bea45240fa86","Type":"ContainerDied","Data":"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"}
Oct 30 18:53:11.616782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.616696    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj" event={"ID":"b5df24f1-8904-4d57-9874-bea45240fa86","Type":"ContainerDied","Data":"2c12eb602efd38fe2995d7848678432453917b53782bb740539410e1e18c8ac3"}
Oct 30 18:53:11.628571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.628554    5065 scope.go:122] "RemoveContainer" containerID="3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"
Oct 30 18:53:11.628888 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:11.628864    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76\": container with ID starting with 3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76 not found: ID does not exist" containerID="3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"
Oct 30 18:53:11.628982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.628897    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76"} err="failed to get container status \"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76\": rpc error: code = NotFound desc = could not find container \"3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76\": container with ID starting with 3fa9adeb8be153696a01c5e7864691351b93b5338ac49b20507f7a11bb739c76 not found: ID does not exist"
Oct 30 18:53:11.628982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.628921    5065 scope.go:122] "RemoveContainer" containerID="129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"
Oct 30 18:53:11.639783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.639766    5065 scope.go:122] "RemoveContainer" containerID="129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"
Oct 30 18:53:11.640066 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:11.640049    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e\": container with ID starting with 129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e not found: ID does not exist" containerID="129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"
Oct 30 18:53:11.640169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.640072    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e"} err="failed to get container status \"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e\": rpc error: code = NotFound desc = could not find container \"129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e\": container with ID starting with 129f80562d2e5963d422f819a5c04031933436e262dfa4736829e9efe9dbd82e not found: ID does not exist"
Oct 30 18:53:11.649849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.649829    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:53:11.654078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.654059    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52fa8b-6d968c7778-dntmc"]
Oct 30 18:53:11.666796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.666771    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:53:11.671478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.671458    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-52fa8b-5f99455455-h6qhj"]
Oct 30 18:53:11.918685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.918613    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" path="/var/lib/kubelet/pods/0e0767d5-b349-45cf-9f6c-47ad7bcadd9c/volumes"
Oct 30 18:53:11.919166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.919143    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320b207b-ad97-46a2-9828-6ec59dc3d08b" path="/var/lib/kubelet/pods/320b207b-ad97-46a2-9828-6ec59dc3d08b/volumes"
Oct 30 18:53:11.919706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.919675    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" path="/var/lib/kubelet/pods/b5df24f1-8904-4d57-9874-bea45240fa86/volumes"
Oct 30 18:53:11.928133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.920778    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de56f31e-f535-4aa3-b14b-65233639d6ed" path="/var/lib/kubelet/pods/de56f31e-f535-4aa3-b14b-65233639d6ed/volumes"
Oct 30 18:53:11.928133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.921453    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e6bdcc-2be8-4a64-aa59-b780e060a1b5" path="/var/lib/kubelet/pods/e7e6bdcc-2be8-4a64-aa59-b780e060a1b5/volumes"
Oct 30 18:53:11.928133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.922056    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640" path="/var/lib/kubelet/pods/f5794ae3-56c7-4fa5-8ed5-4fc97ffdf640/volumes"
Oct 30 18:53:11.986069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.986031    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:53:11.986354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:11.986324    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" podUID="71a525d3-325f-4b1e-a324-591efaf9cd78" containerName="payload-processing" containerID="cri-o://4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0" gracePeriod=30
Oct 30 18:53:12.012734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.012709    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:53:12.012975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.012951    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" podUID="3851f70c-26ab-4766-a9e5-c3080e67fd81" containerName="payload-pre-processing" containerID="cri-o://4fb2fb9bedf3e9cbe8dc021d0d96836bb0486ee555ae62243e13b93c9b2b993c" gracePeriod=30
Oct 30 18:53:12.571803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.543145    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"]
Oct 30 18:53:12.644507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.644475    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d540b-9k6mv"]
Oct 30 18:53:12.658006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.657642    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:53:12.668252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.668224    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" event={"ID":"3851f70c-26ab-4766-a9e5-c3080e67fd81","Type":"ContainerDied","Data":"4fb2fb9bedf3e9cbe8dc021d0d96836bb0486ee555ae62243e13b93c9b2b993c"}
Oct 30 18:53:12.668455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.668436    5065 generic.go:356] "Generic (PLEG): container finished" podID="3851f70c-26ab-4766-a9e5-c3080e67fd81" containerID="4fb2fb9bedf3e9cbe8dc021d0d96836bb0486ee555ae62243e13b93c9b2b993c" exitCode=0
Oct 30 18:53:12.670556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.670533    5065 generic.go:356] "Generic (PLEG): container finished" podID="71a525d3-325f-4b1e-a324-591efaf9cd78" containerID="4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0" exitCode=0
Oct 30 18:53:12.670843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.670773    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" event={"ID":"71a525d3-325f-4b1e-a324-591efaf9cd78","Type":"ContainerDied","Data":"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"}
Oct 30 18:53:12.670973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.670806    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"
Oct 30 18:53:12.671056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.670967    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4" event={"ID":"71a525d3-325f-4b1e-a324-591efaf9cd78","Type":"ContainerDied","Data":"6be04496ac8eebf94aa8dcec9e36138ba03cc926873badc8f7aa69d3523fbb54"}
Oct 30 18:53:12.671207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.671164    5065 scope.go:122] "RemoveContainer" containerID="4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"
Oct 30 18:53:12.704543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.703779    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:53:12.707149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.706738    5065 scope.go:122] "RemoveContainer" containerID="4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"
Oct 30 18:53:12.707149 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:12.707076    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0\": container with ID starting with 4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0 not found: ID does not exist" containerID="4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"
Oct 30 18:53:12.707149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.707136    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0"} err="failed to get container status \"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0\": rpc error: code = NotFound desc = could not find container \"4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0\": container with ID starting with 4b41ea1424739889c0eb7990bdd71befeeb771a39e713e1287c8593735ad1fb0 not found: ID does not exist"
Oct 30 18:53:12.847638 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.847562    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\" (UniqueName: \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\") pod \"3851f70c-26ab-4766-a9e5-c3080e67fd81\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") "
Oct 30 18:53:12.847638 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.847637    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\" (UniqueName: \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\") pod \"71a525d3-325f-4b1e-a324-591efaf9cd78\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") "
Oct 30 18:53:12.847849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.847675    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\") pod \"71a525d3-325f-4b1e-a324-591efaf9cd78\" (UID: \"71a525d3-325f-4b1e-a324-591efaf9cd78\") "
Oct 30 18:53:12.847849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.847828    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\") pod \"3851f70c-26ab-4766-a9e5-c3080e67fd81\" (UID: \"3851f70c-26ab-4766-a9e5-c3080e67fd81\") "
Oct 30 18:53:12.848304 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.848274    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume" pod "71a525d3-325f-4b1e-a324-591efaf9cd78" (UID: "71a525d3-325f-4b1e-a324-591efaf9cd78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:12.848598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.848486    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a525d3-325f-4b1e-a324-591efaf9cd78-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.848705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.848289    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume" pod "3851f70c-26ab-4766-a9e5-c3080e67fd81" (UID: "3851f70c-26ab-4766-a9e5-c3080e67fd81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:12.850033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.850007    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh" pod "3851f70c-26ab-4766-a9e5-c3080e67fd81" (UID: "3851f70c-26ab-4766-a9e5-c3080e67fd81"). InnerVolumeSpecName "kube-api-access-z7nqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:12.850176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.850044    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt" pod "71a525d3-325f-4b1e-a324-591efaf9cd78" (UID: "71a525d3-325f-4b1e-a324-591efaf9cd78"). InnerVolumeSpecName "kube-api-access-2qfnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:12.949713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.949691    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7nqh\" (UniqueName: \"kubernetes.io/projected/3851f70c-26ab-4766-a9e5-c3080e67fd81-kube-api-access-z7nqh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.949713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.949713    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfnt\" (UniqueName: \"kubernetes.io/projected/71a525d3-325f-4b1e-a324-591efaf9cd78-kube-api-access-2qfnt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.949868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:12.949724    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3851f70c-26ab-4766-a9e5-c3080e67fd81-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:13.007745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.007719    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:53:13.019181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.019152    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d540b-bcd8595d4-4xgt4"]
Oct 30 18:53:13.683475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.683443    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww" event={"ID":"3851f70c-26ab-4766-a9e5-c3080e67fd81","Type":"ContainerDied","Data":"0afe6947966724edf5a8f6867147f76dcdd113c46a4a13bd4405fe980f526938"}
Oct 30 18:53:13.683475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.683460    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"
Oct 30 18:53:13.684015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.683512    5065 scope.go:122] "RemoveContainer" containerID="4fb2fb9bedf3e9cbe8dc021d0d96836bb0486ee555ae62243e13b93c9b2b993c"
Oct 30 18:53:13.710936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.710885    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:53:13.716020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.715999    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d540b-7b67d8d655-4vlww"]
Oct 30 18:53:13.900569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.900492    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3851f70c-26ab-4766-a9e5-c3080e67fd81" path="/var/lib/kubelet/pods/3851f70c-26ab-4766-a9e5-c3080e67fd81/volumes"
Oct 30 18:53:13.901543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.901509    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a525d3-325f-4b1e-a324-591efaf9cd78" path="/var/lib/kubelet/pods/71a525d3-325f-4b1e-a324-591efaf9cd78/volumes"
Oct 30 18:53:13.902277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:13.902258    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44cdafd-adca-4c9b-a880-a1266843d54d" path="/var/lib/kubelet/pods/b44cdafd-adca-4c9b-a880-a1266843d54d/volumes"
Oct 30 18:53:14.565101 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.565052    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:53:14.565338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.565309    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" podUID="79f32450-fb33-415d-a973-1bb9c1ba9853" containerName="payload-pre-processing" containerID="cri-o://e5208483e3fd9bc402737fc6e6b9b695abef9a074e491a20987a32c3999999e0" gracePeriod=30
Oct 30 18:53:14.577559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.576580    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:53:14.577559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.576820    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" podUID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" containerName="payload-processing" containerID="cri-o://d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022" gracePeriod=30
Oct 30 18:53:14.693900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.693872    5065 generic.go:356] "Generic (PLEG): container finished" podID="79f32450-fb33-415d-a973-1bb9c1ba9853" containerID="e5208483e3fd9bc402737fc6e6b9b695abef9a074e491a20987a32c3999999e0" exitCode=0
Oct 30 18:53:14.694312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.693958    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" event={"ID":"79f32450-fb33-415d-a973-1bb9c1ba9853","Type":"ContainerDied","Data":"e5208483e3fd9bc402737fc6e6b9b695abef9a074e491a20987a32c3999999e0"}
Oct 30 18:53:14.996990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:14.993675    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"]
Oct 30 18:53:15.000558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.000510    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5ad713-mxlfn"]
Oct 30 18:53:15.112118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.108188    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:53:15.208952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.202830    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:53:15.243421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.243388    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\") pod \"79f32450-fb33-415d-a973-1bb9c1ba9853\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") "
Oct 30 18:53:15.243619 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.243487    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\" (UniqueName: \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\") pod \"79f32450-fb33-415d-a973-1bb9c1ba9853\" (UID: \"79f32450-fb33-415d-a973-1bb9c1ba9853\") "
Oct 30 18:53:15.243869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.243842    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume" pod "79f32450-fb33-415d-a973-1bb9c1ba9853" (UID: "79f32450-fb33-415d-a973-1bb9c1ba9853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:15.244346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.244312    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f32450-fb33-415d-a973-1bb9c1ba9853-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:15.246710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.246686    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z" pod "79f32450-fb33-415d-a973-1bb9c1ba9853" (UID: "79f32450-fb33-415d-a973-1bb9c1ba9853"). InnerVolumeSpecName "kube-api-access-kr97z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:15.345540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.345511    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\") pod \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") "
Oct 30 18:53:15.345763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.345667    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\" (UniqueName: \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\") pod \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\" (UID: \"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef\") "
Oct 30 18:53:15.345970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.345946    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume" pod "3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" (UID: "3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:15.346306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.346285    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr97z\" (UniqueName: \"kubernetes.io/projected/79f32450-fb33-415d-a973-1bb9c1ba9853-kube-api-access-kr97z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:15.346370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.346311    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:15.348080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.348056    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr" pod "3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" (UID: "3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef"). InnerVolumeSpecName "kube-api-access-vnfhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:15.447630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.447603    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfhr\" (UniqueName: \"kubernetes.io/projected/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef-kube-api-access-vnfhr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:15.707594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.707514    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"
Oct 30 18:53:15.707594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.707566    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc" event={"ID":"79f32450-fb33-415d-a973-1bb9c1ba9853","Type":"ContainerDied","Data":"2659ca05706aa98aec11cc4f3cc6efac142099198ba970757f7430bf6b4e4241"}
Oct 30 18:53:15.708101 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.707631    5065 scope.go:122] "RemoveContainer" containerID="e5208483e3fd9bc402737fc6e6b9b695abef9a074e491a20987a32c3999999e0"
Oct 30 18:53:15.709292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.709266    5065 generic.go:356] "Generic (PLEG): container finished" podID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" containerID="d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022" exitCode=0
Oct 30 18:53:15.709431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.709317    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"
Oct 30 18:53:15.709431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.709333    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" event={"ID":"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef","Type":"ContainerDied","Data":"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"}
Oct 30 18:53:15.709431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.709393    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks" event={"ID":"3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef","Type":"ContainerDied","Data":"3a4d898769868fc46fe4edeeec0b8177e27e5e648c99a8bd2a946266c1245392"}
Oct 30 18:53:15.721892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.721876    5065 scope.go:122] "RemoveContainer" containerID="d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"
Oct 30 18:53:15.734071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.733949    5065 scope.go:122] "RemoveContainer" containerID="d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"
Oct 30 18:53:15.734482 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:15.734453    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022\": container with ID starting with d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022 not found: ID does not exist" containerID="d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"
Oct 30 18:53:15.734611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.734492    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022"} err="failed to get container status \"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022\": rpc error: code = NotFound desc = could not find container \"d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022\": container with ID starting with d883947049cf9f08e60b41555a10e0a9942fd38ee774f63104bcb69508bed022 not found: ID does not exist"
Oct 30 18:53:15.742039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.740320    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:53:15.744560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.744539    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-5df449c679-bc5tc"]
Oct 30 18:53:15.759056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.759036    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:53:15.763148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.763129    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5ad713-b99db65d4-tqmks"]
Oct 30 18:53:15.893270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.893236    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" path="/var/lib/kubelet/pods/3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef/volumes"
Oct 30 18:53:15.893847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.893829    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f32450-fb33-415d-a973-1bb9c1ba9853" path="/var/lib/kubelet/pods/79f32450-fb33-415d-a973-1bb9c1ba9853/volumes"
Oct 30 18:53:15.894487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:15.894464    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89423b5c-8a43-40cf-ada8-c16c1a54aaae" path="/var/lib/kubelet/pods/89423b5c-8a43-40cf-ada8-c16c1a54aaae/volumes"
Oct 30 18:53:19.557209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.557175    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"]
Oct 30 18:53:19.571865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.562123    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" containerName="payload-processing"
Oct 30 18:53:19.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.572614    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" containerName="payload-processing"
Oct 30 18:53:19.572822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.572811    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" containerName="payload-pre-processing"
Oct 30 18:53:19.572893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.572883    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" containerName="payload-pre-processing"
Oct 30 18:53:19.572975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.572965    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" containerName="payload-processing"
Oct 30 18:53:19.573035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.573027    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" containerName="payload-processing"
Oct 30 18:53:19.573158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.573149    5065 cpu_manager.go:414] "containerMap: removing container" podUID="71a525d3-325f-4b1e-a324-591efaf9cd78" containerName="payload-processing"
Oct 30 18:53:19.573228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.573220    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a525d3-325f-4b1e-a324-591efaf9cd78" containerName="payload-processing"
Oct 30 18:53:19.574256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.574234    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3851f70c-26ab-4766-a9e5-c3080e67fd81" containerName="payload-pre-processing"
Oct 30 18:53:19.574397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.574384    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3851f70c-26ab-4766-a9e5-c3080e67fd81" containerName="payload-pre-processing"
Oct 30 18:53:19.574514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.574502    5065 cpu_manager.go:414] "containerMap: removing container" podUID="79f32450-fb33-415d-a973-1bb9c1ba9853" containerName="payload-pre-processing"
Oct 30 18:53:19.574606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.574591    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f32450-fb33-415d-a973-1bb9c1ba9853" containerName="payload-pre-processing"
Oct 30 18:53:19.574974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.574959    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a525d3-325f-4b1e-a324-591efaf9cd78" containerName="payload-processing"
Oct 30 18:53:19.575103 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.575071    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3851f70c-26ab-4766-a9e5-c3080e67fd81" containerName="payload-pre-processing"
Oct 30 18:53:19.575200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.575189    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f32450-fb33-415d-a973-1bb9c1ba9853" containerName="payload-pre-processing"
Oct 30 18:53:19.575290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.575281    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3413c424-ffe1-4eaa-8cf2-d4ca2100e4ef" containerName="payload-processing"
Oct 30 18:53:19.575373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.575364    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5df24f1-8904-4d57-9874-bea45240fa86" containerName="payload-pre-processing"
Oct 30 18:53:19.575448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.575439    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0767d5-b349-45cf-9f6c-47ad7bcadd9c" containerName="payload-processing"
Oct 30 18:53:19.611112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.610241    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"]
Oct 30 18:53:19.611112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.610355    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.636109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.635562    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:19.644256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.644229    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:53:19.745129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.745067    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctjn\" (UniqueName: \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.745309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.745159    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.745571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.745555    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.847118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.847018    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctjn\" (UniqueName: \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.847118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.847079    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.847380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.847254    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.847804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.847771    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.849812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.849783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.863643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.863623    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctjn\" (UniqueName: \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:19.965283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:19.965258    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:20.266278 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:20.266251    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2108def3_bd1e_47dd_a0ee_051942541c7b.slice/crio-c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf WatchSource:0}: Error finding container c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf: Status 404 returned error can't find the container with id c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf
Oct 30 18:53:20.266859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:20.266836    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"]
Oct 30 18:53:20.822725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:20.822687    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6" event={"ID":"2108def3-bd1e-47dd-a0ee-051942541c7b","Type":"ContainerStarted","Data":"082c89dc127f91427de82814dd0210be02c6e01f67681d3e9dad4032af0dc986"}
Oct 30 18:53:20.823231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:20.822771    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6" event={"ID":"2108def3-bd1e-47dd-a0ee-051942541c7b","Type":"ContainerStarted","Data":"c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf"}
Oct 30 18:53:20.850790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:20.850739    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6" podStartSLOduration=1.8507285279999999 podStartE2EDuration="1.850728528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:20.849664856 +0000 UTC m=+1221.113129805" watchObservedRunningTime="2026-10-30 18:53:20.850728528 +0000 UTC m=+1221.114193493"
Oct 30 18:53:21.620379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:21.620350    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:21.951070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:21.951026    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:53:21.951886 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:21.951859    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" containerID="cri-o://c712da9fdf340633a883a994b21635edbdd3fcc692d337964ba568df7b369928" gracePeriod=30
Oct 30 18:53:21.976900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:21.976878    5065 generic.go:356] "Generic (PLEG): container finished" podID="2108def3-bd1e-47dd-a0ee-051942541c7b" containerID="082c89dc127f91427de82814dd0210be02c6e01f67681d3e9dad4032af0dc986" exitCode=0
Oct 30 18:53:21.977080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:21.976952    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6" event={"ID":"2108def3-bd1e-47dd-a0ee-051942541c7b","Type":"ContainerDied","Data":"082c89dc127f91427de82814dd0210be02c6e01f67681d3e9dad4032af0dc986"}
Oct 30 18:53:22.489006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.486028    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"]
Oct 30 18:53:22.501973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.501944    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.526160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.526099    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"]
Oct 30 18:53:22.552020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.546861    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 18:53:22.558157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.558072    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.565390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.565361    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 18:53:22.571484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.571461    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.581218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.581195    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 18:53:22.584343 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.584316    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 18:53:22.597973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.597945    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.597989    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hbr\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598113    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598149    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598176    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.598305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598277    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598323    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598369    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598506    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.598560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598535    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8pc\" (UniqueName: \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.598661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.598570    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.699714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699663    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.699871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699733    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.699871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.699871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699834    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.699871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699870    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.699966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700003    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700137    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700135    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700247    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700331    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700364    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700421    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8pc\" (UniqueName: \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.700525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700454    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.700636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700521    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwnk\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.700636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700566    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700587    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700637    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700590    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hbr\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700802    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700860    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700900    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.700969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700929    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.701219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.700994    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.701328 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.701307    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.701583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.701563    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.701824 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.701680    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.703643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.703618    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.704014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.703994    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.704135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.704102    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.718560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.718539    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.718679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.718649    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8pc\" (UniqueName: \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\") pod \"maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.720465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.720444    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hbr\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6ms9r\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.802868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.802841    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.802895    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.802958    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803032    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803135    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803194    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803249    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803297    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803385    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwnk\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803457    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803503 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803385    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.803908 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.803871    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.804026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.804004    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.804309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.804286    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.805622 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.805597    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.805909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.805891    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.816647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.816617    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.820289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.820071    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwnk\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\") pod \"maas-default-gateway-openshift-default-855f9574cf-kwwdj\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:22.866040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.866014    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:22.900016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.899992    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:22.902495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:22.902441    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:23.056076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.055999    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"]
Oct 30 18:53:23.074197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.073142    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0ee2de-f5tc6"]
Oct 30 18:53:23.075416 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:23.075379    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:53:23.183289 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:23.183261    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751bcc56_8b3d_4cc1_86a1_c8fa2d4073c9.slice/crio-63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c WatchSource:0}: Error finding container 63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c: Status 404 returned error can't find the container with id 63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c
Oct 30 18:53:23.184829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.184758    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"]
Oct 30 18:53:23.261338 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:23.261298    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae936fb_f27b_4220_bee1_d985c650d875.slice/crio-12351dceaf880209a1dcaa0c4beea3c06ff58a71476ccc92416e5251ee436d99 WatchSource:0}: Error finding container 12351dceaf880209a1dcaa0c4beea3c06ff58a71476ccc92416e5251ee436d99: Status 404 returned error can't find the container with id 12351dceaf880209a1dcaa0c4beea3c06ff58a71476ccc92416e5251ee436d99
Oct 30 18:53:23.262887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.262860    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 18:53:23.264656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.264624    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.264767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.264698    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.264767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.264741    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.296142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.296114    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 18:53:23.382612 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:23.382584    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487b6a41_850f_4f31_8e00_aedc152e04ee.slice/crio-f30209aa590d0fd30297153e4e7bf54e3761979b355c072ade6f89e6c09838f8 WatchSource:0}: Error finding container f30209aa590d0fd30297153e4e7bf54e3761979b355c072ade6f89e6c09838f8: Status 404 returned error can't find the container with id f30209aa590d0fd30297153e4e7bf54e3761979b355c072ade6f89e6c09838f8
Oct 30 18:53:23.386860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.386319    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.386860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.386413    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.386860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.386460    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:23.457701 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.457676    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:23.518880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.518852    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\" (UniqueName: \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\") pod \"2108def3-bd1e-47dd-a0ee-051942541c7b\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") "
Oct 30 18:53:23.522545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.522512    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn" pod "2108def3-bd1e-47dd-a0ee-051942541c7b" (UID: "2108def3-bd1e-47dd-a0ee-051942541c7b"). InnerVolumeSpecName "kube-api-access-9ctjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.522856    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\") pod \"2108def3-bd1e-47dd-a0ee-051942541c7b\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") "
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.523476    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\") pod \"2108def3-bd1e-47dd-a0ee-051942541c7b\" (UID: \"2108def3-bd1e-47dd-a0ee-051942541c7b\") "
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.523878    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca" pod "2108def3-bd1e-47dd-a0ee-051942541c7b" (UID: "2108def3-bd1e-47dd-a0ee-051942541c7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.524657    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.524679    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctjn\" (UniqueName: \"kubernetes.io/projected/2108def3-bd1e-47dd-a0ee-051942541c7b-kube-api-access-9ctjn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:23.525880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.525341    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca" pod "2108def3-bd1e-47dd-a0ee-051942541c7b" (UID: "2108def3-bd1e-47dd-a0ee-051942541c7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:23.626497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.626394    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108def3-bd1e-47dd-a0ee-051942541c7b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:23.899146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:23.899038    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5" path="/var/lib/kubelet/pods/9ffa2d77-c1bd-4d34-bba4-23d9d5c04bd5/volumes"
Oct 30 18:53:24.051069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.051032    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" event={"ID":"487b6a41-850f-4f31-8e00-aedc152e04ee","Type":"ContainerStarted","Data":"668b07d0b94c45d04c0cbc3de938bf30b87c850a9427c8b355ee6a6b88ec598f"}
Oct 30 18:53:24.051245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.051138    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" event={"ID":"487b6a41-850f-4f31-8e00-aedc152e04ee","Type":"ContainerStarted","Data":"f30209aa590d0fd30297153e4e7bf54e3761979b355c072ade6f89e6c09838f8"}
Oct 30 18:53:24.052734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.052706    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"
Oct 30 18:53:24.052870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.052751    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6" event={"ID":"2108def3-bd1e-47dd-a0ee-051942541c7b","Type":"ContainerDied","Data":"c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf"}
Oct 30 18:53:24.052870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.052791    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41950ef67c7e5376f788a7285d5bacb8a4a71457d2c38b9f10c359c593bafaf"
Oct 30 18:53:24.054746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.054669    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" event={"ID":"1ae936fb-f27b-4220-bee1-d985c650d875","Type":"ContainerStarted","Data":"920d135993d43415e3069f29532bc8d4189279dbaf4eac5a4b64131f43bf1a4d"}
Oct 30 18:53:24.054746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.054714    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" event={"ID":"1ae936fb-f27b-4220-bee1-d985c650d875","Type":"ContainerStarted","Data":"12351dceaf880209a1dcaa0c4beea3c06ff58a71476ccc92416e5251ee436d99"}
Oct 30 18:53:24.056722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.056704    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z" event={"ID":"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9","Type":"ContainerStarted","Data":"57cd131bb168ec2f1a567902cc05a791d1da1101039e316088e7806207c58321"}
Oct 30 18:53:24.057032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.056742    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z" event={"ID":"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9","Type":"ContainerStarted","Data":"63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c"}
Oct 30 18:53:24.076959 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.076917    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podStartSLOduration=2.076905099 podStartE2EDuration="2.076905099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:24.076564164 +0000 UTC m=+1224.340029114" watchObservedRunningTime="2026-10-30 18:53:24.076905099 +0000 UTC m=+1224.340370048"
Oct 30 18:53:24.106768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.106721    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z" podStartSLOduration=2.106708543 podStartE2EDuration="2.106708543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:24.104989801 +0000 UTC m=+1224.368454768" watchObservedRunningTime="2026-10-30 18:53:24.106708543 +0000 UTC m=+1224.370173520"
Oct 30 18:53:24.127323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.127272    5065 patch_prober.go:29] interesting pod/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:53:24.127462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.127333    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:24.137594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.137500    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podStartSLOduration=2.137482256 podStartE2EDuration="2.137482256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:24.130617981 +0000 UTC m=+1224.394082930" watchObservedRunningTime="2026-10-30 18:53:24.137482256 +0000 UTC m=+1224.400947207"
Oct 30 18:53:24.908924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.906931    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:24.909728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.909701    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:24.909927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.909904    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:24.957201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.957074    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:24.959255 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.959228    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:24.959515 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:24.959482    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:25.296112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.285348    5065 generic.go:356] "Generic (PLEG): container finished" podID="751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" containerID="57cd131bb168ec2f1a567902cc05a791d1da1101039e316088e7806207c58321" exitCode=0
Oct 30 18:53:25.319142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.293815    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z" event={"ID":"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9","Type":"ContainerDied","Data":"57cd131bb168ec2f1a567902cc05a791d1da1101039e316088e7806207c58321"}
Oct 30 18:53:25.649436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.649359    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"]
Oct 30 18:53:25.650406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.650386    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2108def3-bd1e-47dd-a0ee-051942541c7b" containerName="revoke-keys"
Oct 30 18:53:25.650540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.650527    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2108def3-bd1e-47dd-a0ee-051942541c7b" containerName="revoke-keys"
Oct 30 18:53:25.650755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.650745    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2108def3-bd1e-47dd-a0ee-051942541c7b" containerName="revoke-keys"
Oct 30 18:53:25.654535 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.654406    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"]
Oct 30 18:53:25.654707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.654508    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:25.835102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.835043    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:25.835388 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.835365    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2c7\" (UniqueName: \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:25.835570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.835535    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:25.900789 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.900688    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:25.909134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.900977    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:26.028264 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.994793    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:26.032753 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.032703    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:26.033045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:25.987832    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.033945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.033607    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.043262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.043239    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2c7\" (UniqueName: \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.046614 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.013292    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.133967 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.133933    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2c7\" (UniqueName: \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.147623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.147594    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.168164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.168053    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"]
Oct 30 18:53:26.169765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.169742    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb394f71-zjxhj"]
Oct 30 18:53:26.275869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.275833    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:26.871256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.871203    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:26.953732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.951834    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:26.953732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.951935    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:26.953732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.952395    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:26.953732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.952432    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:26.973465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.973438    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\") pod \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") "
Oct 30 18:53:26.974528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.974490    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\") pod \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") "
Oct 30 18:53:26.974753 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:26.973540    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1afd6b_0a92_418b_99b7_f8a4c84c1af7.slice/crio-8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0 WatchSource:0}: Error finding container 8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0: Status 404 returned error can't find the container with id 8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0
Oct 30 18:53:26.977297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.975340    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca" pod "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" (UID: "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:26.977297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.977190    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca" pod "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" (UID: "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:26.977695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.977661    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\" (UniqueName: \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\") pod \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\" (UID: \"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9\") "
Oct 30 18:53:26.991951 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.991930    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:26.992114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:26.992077    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.034149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.020280    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc" pod "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" (UID: "751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9"). InnerVolumeSpecName "kube-api-access-4c8pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:27.055268 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.053274    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"]
Oct 30 18:53:27.109489 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.108687    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8pc\" (UniqueName: \"kubernetes.io/projected/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9-kube-api-access-4c8pc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.550075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.550039    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z" event={"ID":"751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9","Type":"ContainerDied","Data":"63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c"}
Oct 30 18:53:27.550387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.550370    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a4f6fdded7ddd45af4007303ee2503808ab94196bd2f5bdd60c18b3749e29c"
Oct 30 18:53:27.550554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.550542    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"
Oct 30 18:53:27.561451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.561423    5065 generic.go:356] "Generic (PLEG): container finished" podID="3252a0df-cb7e-49ec-b5cb-330415724228" containerID="c712da9fdf340633a883a994b21635edbdd3fcc692d337964ba568df7b369928" exitCode=0
Oct 30 18:53:27.561731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.561711    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerDied","Data":"c712da9fdf340633a883a994b21635edbdd3fcc692d337964ba568df7b369928"}
Oct 30 18:53:27.561882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.561869    5065 scope.go:122] "RemoveContainer" containerID="1b8817c90a28993438df22460dc52be021f6f224c4fb333315ad5c19db73e79f"
Oct 30 18:53:27.564686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.564661    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk" event={"ID":"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7","Type":"ContainerStarted","Data":"8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0"}
Oct 30 18:53:27.677697 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.677581    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:53:27.867321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867278    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867354    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867490    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867521    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867584    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867667    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867732    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867762    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.867938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867832    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:27.867938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.867922    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\") pod \"3252a0df-cb7e-49ec-b5cb-330415724228\" (UID: \"3252a0df-cb7e-49ec-b5cb-330415724228\") "
Oct 30 18:53:27.868060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868028    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:27.868492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868457    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:27.868624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868478    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:27.868962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868939    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.869053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868980    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3252a0df-cb7e-49ec-b5cb-330415724228-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.869053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.868996    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.869053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.869010    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.869543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.869516    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:27.871174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.871145    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:27.871322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.871199    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "kube-api-access-bj5bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:27.871645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.871591    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:27.871957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.871932    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy" pod "3252a0df-cb7e-49ec-b5cb-330415724228" (UID: "3252a0df-cb7e-49ec-b5cb-330415724228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:27.909108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.901054    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:27.909108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.901129    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:27.944577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.943424    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:27.944577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.943479    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:27.956144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.956116    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f15acf8-d3d8-47f9-b39f-73b172ca1d81" path="/var/lib/kubelet/pods/0f15acf8-d3d8-47f9-b39f-73b172ca1d81/volumes"
Oct 30 18:53:27.970012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.969991    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.970147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.970015    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3252a0df-cb7e-49ec-b5cb-330415724228-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.970147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.970031    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5bp\" (UniqueName: \"kubernetes.io/projected/3252a0df-cb7e-49ec-b5cb-330415724228-kube-api-access-bj5bp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.970147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.970048    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:27.970147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:27.970062    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3252a0df-cb7e-49ec-b5cb-330415724228-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:28.574999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.574962    5065 generic.go:356] "Generic (PLEG): container finished" podID="6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" containerID="b1413ac2739bd0525425594f02c873956c3ff19b92c21c0a4ae8c5bf6c9934e3" exitCode=0
Oct 30 18:53:28.575182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.575045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk" event={"ID":"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7","Type":"ContainerDied","Data":"b1413ac2739bd0525425594f02c873956c3ff19b92c21c0a4ae8c5bf6c9934e3"}
Oct 30 18:53:28.576751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.576726    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg" event={"ID":"3252a0df-cb7e-49ec-b5cb-330415724228","Type":"ContainerDied","Data":"d9d1142977c48fe3dfd500237eb3db8bf3bdf16843bd782b8c62d80c2a30b980"}
Oct 30 18:53:28.576901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.576775    5065 scope.go:122] "RemoveContainer" containerID="c712da9fdf340633a883a994b21635edbdd3fcc692d337964ba568df7b369928"
Oct 30 18:53:28.576901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.576737    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"
Oct 30 18:53:28.653230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.653200    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:53:28.658894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.658872    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0d540b-openshift-default-796b4bf69-2mhtg"]
Oct 30 18:53:28.926672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.926455    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:28.926672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.926522    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:28.926672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.926559    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:28.926672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:28.926601    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:29.974752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:29.974661    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:29.975339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:29.975307    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:30.074113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.073638    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:30.074113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.073708    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:30.153975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.153940    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" path="/var/lib/kubelet/pods/3252a0df-cb7e-49ec-b5cb-330415724228/volumes"
Oct 30 18:53:30.270839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.270812    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:30.378146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.378025    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\") pod \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") "
Oct 30 18:53:30.378146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.378147    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\" (UniqueName: \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\") pod \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") "
Oct 30 18:53:30.378362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.378226    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\") pod \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\" (UID: \"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7\") "
Oct 30 18:53:30.387127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.379146    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca" pod "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" (UID: "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:30.438844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.435159    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7" pod "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" (UID: "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7"). InnerVolumeSpecName "kube-api-access-nc2c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:30.438844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.438517    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca" pod "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" (UID: "6a1afd6b-0a92-418b-99b7-f8a4c84c1af7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:30.479916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.479852    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.480182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.480161    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc2c7\" (UniqueName: \"kubernetes.io/projected/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-kube-api-access-nc2c7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.480182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.480187    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.629162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.627232    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk" event={"ID":"6a1afd6b-0a92-418b-99b7-f8a4c84c1af7","Type":"ContainerDied","Data":"8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0"}
Oct 30 18:53:30.629162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.627283    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"
Oct 30 18:53:30.629162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.627295    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8543812afaa21a9db9174007f8012ab3dbc798fb6c75f1a04f3270507396d6e0"
Oct 30 18:53:30.908200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.908076    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:30.908200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.908160    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:30.908574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.908554    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:30.908615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:30.908593    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:31.902916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:31.902867    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6ms9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 18:53:31.903372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:31.902935    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 18:53:31.903455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:31.903405    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 18:53:31.903509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:31.903458    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 18:53:32.795291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:32.795255    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:33.914137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.909346    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:33.914137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.911140    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:33.914137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.912961    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kwwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:33.914137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.913013    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:33.914137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.913667    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 18:53:33.971114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.970198    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:33.971342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.971174    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:33.973564 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:33.973537    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 18:53:34.910112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:34.909160    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"]
Oct 30 18:53:34.914232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:34.914205    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45f649-s9s9z"]
Oct 30 18:53:35.102231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.102191    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"]
Oct 30 18:53:35.174260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.174169    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"]
Oct 30 18:53:35.185746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.185720    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a0b435-48plv"]
Oct 30 18:53:35.189042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.189016    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95191225-vm22g"]
Oct 30 18:53:35.890864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.890832    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da829f1-d552-4728-af0d-543a90b19ae3" path="/var/lib/kubelet/pods/0da829f1-d552-4728-af0d-543a90b19ae3/volumes"
Oct 30 18:53:35.891319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.891305    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" path="/var/lib/kubelet/pods/751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9/volumes"
Oct 30 18:53:35.891710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:35.891697    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf729ee1-6872-4021-a9db-033f5aca60e1" path="/var/lib/kubelet/pods/bf729ee1-6872-4021-a9db-033f5aca60e1/volumes"
Oct 30 18:53:37.796624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:37.796574    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" podUID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" containerName="payload-pre-processing" containerID="cri-o://1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb" gracePeriod=25
Oct 30 18:53:38.047954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.047882    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:53:38.049194 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.048430    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" podUID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" containerName="payload-processing" containerID="cri-o://8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad" gracePeriod=30
Oct 30 18:53:38.057726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.057644    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:53:38.058014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.057865    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" podUID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" containerName="payload-pre-processing" containerID="cri-o://cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e" gracePeriod=30
Oct 30 18:53:38.271813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.271400    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:38.479250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.469435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\") pod \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") "
Oct 30 18:53:38.479250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.469563    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\" (UniqueName: \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\") pod \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\" (UID: \"8d624ad4-d108-4ec6-a60b-9e112960bb3f\") "
Oct 30 18:53:38.479250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.471375    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume" pod "8d624ad4-d108-4ec6-a60b-9e112960bb3f" (UID: "8d624ad4-d108-4ec6-a60b-9e112960bb3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:38.479250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.475281    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"]
Oct 30 18:53:38.490444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.483127    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fd4ed-tmhd6"]
Oct 30 18:53:38.490444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.483171    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk" pod "8d624ad4-d108-4ec6-a60b-9e112960bb3f" (UID: "8d624ad4-d108-4ec6-a60b-9e112960bb3f"). InnerVolumeSpecName "kube-api-access-d6qkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:38.571992 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.571914    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d624ad4-d108-4ec6-a60b-9e112960bb3f-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.571992 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.571953    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qkk\" (UniqueName: \"kubernetes.io/projected/8d624ad4-d108-4ec6-a60b-9e112960bb3f-kube-api-access-d6qkk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.685110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.675512    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:53:38.711259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.711219    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:53:38.774910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.774248    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\") pod \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") "
Oct 30 18:53:38.774910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.774373    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\" (UniqueName: \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\") pod \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\" (UID: \"e2b6aeb7-8c81-46a0-82b6-252f64d39492\") "
Oct 30 18:53:38.774910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.774654    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume" pod "e2b6aeb7-8c81-46a0-82b6-252f64d39492" (UID: "e2b6aeb7-8c81-46a0-82b6-252f64d39492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:38.775174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.774999    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b6aeb7-8c81-46a0-82b6-252f64d39492-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.777235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.777213    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9" pod "e2b6aeb7-8c81-46a0-82b6-252f64d39492" (UID: "e2b6aeb7-8c81-46a0-82b6-252f64d39492"). InnerVolumeSpecName "kube-api-access-nmjs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:38.824908 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.824863    5065 generic.go:356] "Generic (PLEG): container finished" podID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" containerID="cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e" exitCode=0
Oct 30 18:53:38.825287 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.824942    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" event={"ID":"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb","Type":"ContainerDied","Data":"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"}
Oct 30 18:53:38.825355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.825294    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx" event={"ID":"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb","Type":"ContainerDied","Data":"4614915f0c3a757f2b2f21fe2e40398f7098c83a94c7cccc58c8904f00717ebe"}
Oct 30 18:53:38.825355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.825331    5065 scope.go:122] "RemoveContainer" containerID="cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"
Oct 30 18:53:38.825460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.824982    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"
Oct 30 18:53:38.828026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.828003    5065 generic.go:356] "Generic (PLEG): container finished" podID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" containerID="8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad" exitCode=0
Oct 30 18:53:38.833131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.828302    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" event={"ID":"e2b6aeb7-8c81-46a0-82b6-252f64d39492","Type":"ContainerDied","Data":"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"}
Oct 30 18:53:38.833131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.828349    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k" event={"ID":"e2b6aeb7-8c81-46a0-82b6-252f64d39492","Type":"ContainerDied","Data":"6998a11869e800f6bd4f7444ed8028206ae2cd0e651d37a10ea68eea5a52309a"}
Oct 30 18:53:38.833131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.828421    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"
Oct 30 18:53:38.833131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.833020    5065 generic.go:356] "Generic (PLEG): container finished" podID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" containerID="1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb" exitCode=0
Oct 30 18:53:38.833387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.833224    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" event={"ID":"8d624ad4-d108-4ec6-a60b-9e112960bb3f","Type":"ContainerDied","Data":"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"}
Oct 30 18:53:38.833387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.833255    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r" event={"ID":"8d624ad4-d108-4ec6-a60b-9e112960bb3f","Type":"ContainerDied","Data":"ce4d425dfc51a04799ab16b7776a98ea5d3259ecb22df1a8824b52e24657679f"}
Oct 30 18:53:38.833387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.833305    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"
Oct 30 18:53:38.843253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.843236    5065 scope.go:122] "RemoveContainer" containerID="cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"
Oct 30 18:53:38.843682 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:38.843637    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e\": container with ID starting with cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e not found: ID does not exist" containerID="cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"
Oct 30 18:53:38.843767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.843703    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e"} err="failed to get container status \"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e\": rpc error: code = NotFound desc = could not find container \"cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e\": container with ID starting with cf5d9aa197182441f3d5837ec95754d83704dc45ec94ff2329f3755bac838b8e not found: ID does not exist"
Oct 30 18:53:38.843767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.843727    5065 scope.go:122] "RemoveContainer" containerID="8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"
Oct 30 18:53:38.865938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.865914    5065 scope.go:122] "RemoveContainer" containerID="8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"
Oct 30 18:53:38.866331 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:38.866306    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad\": container with ID starting with 8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad not found: ID does not exist" containerID="8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"
Oct 30 18:53:38.866431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.866342    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad"} err="failed to get container status \"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad\": rpc error: code = NotFound desc = could not find container \"8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad\": container with ID starting with 8610eb96c16c5a72f5c9582905b8329eb0144b7cb245d4767f6656bec1890bad not found: ID does not exist"
Oct 30 18:53:38.866431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.866366    5065 scope.go:122] "RemoveContainer" containerID="1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"
Oct 30 18:53:38.868764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.868726    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:53:38.873072 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.873051    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-fd6d54466-lvz9k"]
Oct 30 18:53:38.880879 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.880856    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\" (UniqueName: \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\") pod \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") "
Oct 30 18:53:38.880984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.880900    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\") pod \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\" (UID: \"addfa4fc-47b6-4b81-acfb-2b8e118f6ecb\") "
Oct 30 18:53:38.881620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.881337    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume" pod "addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" (UID: "addfa4fc-47b6-4b81-acfb-2b8e118f6ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:38.881794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.881742    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.881794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.881766    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjs9\" (UniqueName: \"kubernetes.io/projected/e2b6aeb7-8c81-46a0-82b6-252f64d39492-kube-api-access-nmjs9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.881974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.881953    5065 scope.go:122] "RemoveContainer" containerID="1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"
Oct 30 18:53:38.882328 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:38.882311    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb\": container with ID starting with 1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb not found: ID does not exist" containerID="1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"
Oct 30 18:53:38.882394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.882335    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb"} err="failed to get container status \"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb\": rpc error: code = NotFound desc = could not find container \"1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb\": container with ID starting with 1b3da8f1cb9247cd2cb1f0a561fd99d332bad506dd4e4de08a3957c93f479ccb not found: ID does not exist"
Oct 30 18:53:38.883519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.883491    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g" pod "addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" (UID: "addfa4fc-47b6-4b81-acfb-2b8e118f6ecb"). InnerVolumeSpecName "kube-api-access-njh5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:38.892625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.892604    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:38.910615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.906696    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-55fc578db7-w9m4r"]
Oct 30 18:53:38.983467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:38.983434    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njh5g\" (UniqueName: \"kubernetes.io/projected/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb-kube-api-access-njh5g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:39.154078 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.154049    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:53:39.166801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.166778    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-6cdbbd7b55-nj5vx"]
Oct 30 18:53:39.892045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.892010    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2108def3-bd1e-47dd-a0ee-051942541c7b" path="/var/lib/kubelet/pods/2108def3-bd1e-47dd-a0ee-051942541c7b/volumes"
Oct 30 18:53:39.892631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.892613    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" path="/var/lib/kubelet/pods/8d624ad4-d108-4ec6-a60b-9e112960bb3f/volumes"
Oct 30 18:53:39.893167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.893149    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" path="/var/lib/kubelet/pods/addfa4fc-47b6-4b81-acfb-2b8e118f6ecb/volumes"
Oct 30 18:53:39.894108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:39.894073    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" path="/var/lib/kubelet/pods/e2b6aeb7-8c81-46a0-82b6-252f64d39492/volumes"
Oct 30 18:53:42.055406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:42.055373    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"]
Oct 30 18:53:42.058914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:42.058889    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecef08-qf4gk"]
Oct 30 18:53:43.895068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:43.895032    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" path="/var/lib/kubelet/pods/6a1afd6b-0a92-418b-99b7-f8a4c84c1af7/volumes"
Oct 30 18:53:44.026226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:44.025984    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:53:44.026498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:44.026462    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy" containerID="cri-o://5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c" gracePeriod=30
Oct 30 18:53:46.260322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.260285    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:53:46.261265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261240    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:46.261265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261265    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261295    5065 cpu_manager.go:414] "containerMap: removing container" podUID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" containerName="payload-pre-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261303    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" containerName="payload-pre-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261336    5065 cpu_manager.go:414] "containerMap: removing container" podUID="751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" containerName="revoke-keys"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261344    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" containerName="revoke-keys"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261358    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" containerName="payload-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261365    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" containerName="payload-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261405    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" containerName="payload-pre-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261413    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" containerName="payload-pre-processing"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261429    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" containerName="revoke-keys"
Oct 30 18:53:46.261463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261436    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" containerName="revoke-keys"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261601    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d624ad4-d108-4ec6-a60b-9e112960bb3f" containerName="payload-pre-processing"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261614    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1afd6b-0a92-418b-99b7-f8a4c84c1af7" containerName="revoke-keys"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261630    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261641    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261652    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="addfa4fc-47b6-4b81-acfb-2b8e118f6ecb" containerName="payload-pre-processing"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261664    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="751bcc56-8b3d-4cc1-86a1-c8fa2d4073c9" containerName="revoke-keys"
Oct 30 18:53:46.261953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.261673    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b6aeb7-8c81-46a0-82b6-252f64d39492" containerName="payload-processing"
Oct 30 18:53:46.265438 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.265417    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.301018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.300994    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fa7b20-openshift-default-dockercfg-44x6z\""
Oct 30 18:53:46.367125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.359506    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:53:46.374202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.367413    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.374507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.374487    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.374691 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.374672    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.374898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.374877    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.375057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.375041    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.375310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.375292    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.375557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.375537    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xz6g\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.375865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.375848    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.376009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.375995    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.492908    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.492968    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493039    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493105    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493136    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493219    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493259    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.493326    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xz6g\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.494167    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.494407    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.521170 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.510889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.553112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.547494    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.553112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.547796    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.632108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.622505    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xz6g\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.632108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.625133    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.632108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.625844    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.632108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.626263    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\") pod \"e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") " pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:46.879993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:46.879904    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:47.238763 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:53:47.238723    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79044e03_a475_4595_b8f3_04e6866d9558.slice/crio-957b3cb4ae2a7c06fcca30f384c83dbc0afa3d9e518d7178c13125c2d2459879 WatchSource:0}: Error finding container 957b3cb4ae2a7c06fcca30f384c83dbc0afa3d9e518d7178c13125c2d2459879: Status 404 returned error can't find the container with id 957b3cb4ae2a7c06fcca30f384c83dbc0afa3d9e518d7178c13125c2d2459879
Oct 30 18:53:47.239460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:47.239440    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:53:47.241223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:47.241196    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:47.241292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:47.241259    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:47.241329 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:47.241290    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:48.093803 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:48.093760    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerStarted","Data":"3f1cf3ab5cdaf3d95f421d0a6e322f4b2235656a08bfdee41300a637226b329a"}
Oct 30 18:53:48.094323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:48.093826    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerStarted","Data":"957b3cb4ae2a7c06fcca30f384c83dbc0afa3d9e518d7178c13125c2d2459879"}
Oct 30 18:53:48.131463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:48.131397    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podStartSLOduration=2.131377453 podStartE2EDuration="2.131377453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:48.122107369 +0000 UTC m=+1248.385572329" watchObservedRunningTime="2026-10-30 18:53:48.131377453 +0000 UTC m=+1248.394842402"
Oct 30 18:53:48.882119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:48.882007    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:49.001507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.001465    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:49.118225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.117650    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:49.118881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.118859    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:53:49.377766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.376296    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:53:49.464165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.463600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.464165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.463665    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.464165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.463770    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.464165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.463818    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475348    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475496    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475536    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475695    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.475853    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\") pod \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\" (UID: \"4f3d595a-e81f-4461-abd4-4a8174a11c1b\") "
Oct 30 18:53:49.476601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.476574    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.481166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.480312    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:49.481166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.480957    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:49.489111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.488189    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:49.490417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.490392    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:49.522110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.507615    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:49.532398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.532222    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:49.532398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.532316    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:49.545111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.535420    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt" pod "4f3d595a-e81f-4461-abd4-4a8174a11c1b" (UID: "4f3d595a-e81f-4461-abd4-4a8174a11c1b"). InnerVolumeSpecName "kube-api-access-8mvnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578823    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578863    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578876    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578887    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578898    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578922    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvnt\" (UniqueName: \"kubernetes.io/projected/4f3d595a-e81f-4461-abd4-4a8174a11c1b-kube-api-access-8mvnt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578933    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3d595a-e81f-4461-abd4-4a8174a11c1b-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:49.582310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:49.578944    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3d595a-e81f-4461-abd4-4a8174a11c1b-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:50.059529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.059493    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"]
Oct 30 18:53:50.063206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.063185    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d062c581-spdl8"]
Oct 30 18:53:50.127993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.127955    5065 generic.go:356] "Generic (PLEG): container finished" podID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerID="5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c" exitCode=0
Oct 30 18:53:50.128378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.128009    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"
Oct 30 18:53:50.128378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.128044    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerDied","Data":"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"}
Oct 30 18:53:50.128378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.128146    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk" event={"ID":"4f3d595a-e81f-4461-abd4-4a8174a11c1b","Type":"ContainerDied","Data":"d3b913c2867155e0c42dcb006286c5e045cef39bca8a9a4fdc9988d93240e096"}
Oct 30 18:53:50.128378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.128178    5065 scope.go:122] "RemoveContainer" containerID="5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"
Oct 30 18:53:50.140780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.140761    5065 scope.go:122] "RemoveContainer" containerID="8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"
Oct 30 18:53:50.151838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.151817    5065 scope.go:122] "RemoveContainer" containerID="5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"
Oct 30 18:53:50.152116 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:50.152066    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c\": container with ID starting with 5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c not found: ID does not exist" containerID="5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"
Oct 30 18:53:50.152191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.152121    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c"} err="failed to get container status \"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c\": rpc error: code = NotFound desc = could not find container \"5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c\": container with ID starting with 5c6c7e61de164e7ac14481591922790390e49ca005928efffacc20d1cbb0280c not found: ID does not exist"
Oct 30 18:53:50.152191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.152140    5065 scope.go:122] "RemoveContainer" containerID="8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"
Oct 30 18:53:50.152436 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:50.152416    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693\": container with ID starting with 8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693 not found: ID does not exist" containerID="8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"
Oct 30 18:53:50.152483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.152443    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693"} err="failed to get container status \"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693\": rpc error: code = NotFound desc = could not find container \"8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693\": container with ID starting with 8eb0928da6eedc777c0fe0df3832283452c6a00c2bba720ecd292ef2bf2f6693 not found: ID does not exist"
Oct 30 18:53:50.173756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.173735    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:53:50.177979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:50.177960    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45f649-openshift-default-59869958d9-dnlfk"]
Oct 30 18:53:51.566360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.566330    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:53:51.567363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567343    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.567363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567363    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.567547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567393    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:51.567547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567401    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3252a0df-cb7e-49ec-b5cb-330415724228" containerName="istio-proxy"
Oct 30 18:53:51.567547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567428    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.567547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567437    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.567764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567629    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.567764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.567647    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" containerName="istio-proxy"
Oct 30 18:53:51.576656 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.576629    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.584455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.578700    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:53:51.617107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.614547    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f2ceb5-openshift-default-dockercfg-kk9sl\""
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625292    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625367    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625422    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625458    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625496    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625574    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625621    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625655    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.626157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.625722    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbll\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751416    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751546    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751605    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751639    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751730    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbll\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751867    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751923    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.751987    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.753122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.752039    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.791111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.758383    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.791111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.758662    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.791111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.759271    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.791111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.759523    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.791111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.759810    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.803112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.796197    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.803112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.796704    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.815973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.815933    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.826576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.826469    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbll\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\") pod \"e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") " pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:51.953680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.952275    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbfe700-0eee-4dcd-a1c0-862abd70a4a5" path="/var/lib/kubelet/pods/0fbfe700-0eee-4dcd-a1c0-862abd70a4a5/volumes"
Oct 30 18:53:51.953680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.953050    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3d595a-e81f-4461-abd4-4a8174a11c1b" path="/var/lib/kubelet/pods/4f3d595a-e81f-4461-abd4-4a8174a11c1b/volumes"
Oct 30 18:53:51.976703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:51.976628    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:52.344743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:52.344668    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:53:52.347765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:52.347731    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:52.347878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:52.347833    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:52.347950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:52.347876    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:53:53.094191 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:53:53.094150    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:53:53.324786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:53.324749    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerStarted","Data":"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"}
Oct 30 18:53:53.324977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:53.324818    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerStarted","Data":"f2d89ec2601e1abf9731d0b099f8909b2a92fad44d2f2c7d7c5a26e330b4f94a"}
Oct 30 18:53:53.358188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:53.358061    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podStartSLOduration=2.358048256 podStartE2EDuration="2.358048256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:53.354955011 +0000 UTC m=+1253.618419958" watchObservedRunningTime="2026-10-30 18:53:53.358048256 +0000 UTC m=+1253.621513204"
Oct 30 18:53:53.976833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:53.976800    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:53.982205 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:53.982180    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:54.334845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:54.334819    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:54.335887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:54.335867    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:53:59.001050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.001019    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:53:59.007208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.007186    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.020119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.020080    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:53:59.118443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.118407    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d5ab51\""
Oct 30 18:53:59.118901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.118881    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d5ab51-dockercfg-lz8xd\""
Oct 30 18:53:59.216293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.198546    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.216293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.210289    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgsn\" (UniqueName: \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.344688 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.344628    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.344972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.344946    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgsn\" (UniqueName: \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.348423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.348331    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.365011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.364981    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgsn\" (UniqueName: \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\") pod \"payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:53:59.641335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:53:59.641253    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:54:00.106826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:00.106794    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:54:00.422917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:00.422870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" event={"ID":"0f67c87c-2473-4870-9d0a-3039d136b7ae","Type":"ContainerStarted","Data":"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"}
Oct 30 18:54:00.423166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:00.422933    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" event={"ID":"0f67c87c-2473-4870-9d0a-3039d136b7ae","Type":"ContainerStarted","Data":"7a00489fe1ce8f7fd9b73ff8038625c4673dcd0e4016cf3d34b3b3e049b9a121"}
Oct 30 18:54:00.423166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:00.422975    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:54:00.447716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:00.447650    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" podStartSLOduration=2.447635873 podStartE2EDuration="2.447635873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:00.443515156 +0000 UTC m=+1260.706980153" watchObservedRunningTime="2026-10-30 18:54:00.447635873 +0000 UTC m=+1260.711100837"
Oct 30 18:54:02.515698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.515661    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 18:54:02.637834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.637808    5065 scope.go:122] "RemoveContainer" containerID="c3478bdaa5f842103768e837e885ac070c3d1e89b20ac076bd32966d651d1ca4"
Oct 30 18:54:02.651940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.651919    5065 scope.go:122] "RemoveContainer" containerID="73b1ccdc407c87388670e14f2ed18187a7729033ab3b91820aaa339617e1d0b9"
Oct 30 18:54:02.665994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.665970    5065 scope.go:122] "RemoveContainer" containerID="18642b955dc67bff3d7864927486d34f6aebb74a5589f1fbc9394bb97ad86c8b"
Oct 30 18:54:02.676620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.676601    5065 scope.go:122] "RemoveContainer" containerID="f065cf72381805abceee3e4cfe3a1014875c9cf6e626d552e38150ac26426a02"
Oct 30 18:54:02.686353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.686336    5065 scope.go:122] "RemoveContainer" containerID="2097e77c1312c6076749b1ecac9999ec9f4714db9766c2e3e292bff59cdf6291"
Oct 30 18:54:02.696305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.696284    5065 scope.go:122] "RemoveContainer" containerID="719f900f39e5f49662b26163f4a9350a7e9102f0f7e35cd39be954b5a2e834ed"
Oct 30 18:54:02.706646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.706620    5065 scope.go:122] "RemoveContainer" containerID="964cfd3be904bbf2997d426123ceb79222d46fc413165e3c16e3f6050bdded50"
Oct 30 18:54:02.716602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.716578    5065 scope.go:122] "RemoveContainer" containerID="266e2c50563c60d41b3adfcdf8750d14dc37775d0e6f65c050f6712bc031e434"
Oct 30 18:54:02.735910 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:02.735893    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 18:54:10.702112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.698159    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:54:10.702112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.698425    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy" containerID="cri-o://a130c48dfc6ce92efd37887f41d72052c581f8482ff6ef61da1154d9e9465bb6" gracePeriod=30
Oct 30 18:54:10.939173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.939136    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:54:10.940884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.939782    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy" containerID="cri-o://4303b046109983061a4ad82eb713dc3f3989753a2322d0eaa0b41d9c698d99b1" gracePeriod=30
Oct 30 18:54:10.947163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.947137    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:54:10.947385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:10.947323    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy" containerID="cri-o://b5ab877112db8a14f71c669e38bc4e0650d238d605674ff5c1d50a00bad09f24" gracePeriod=30
Oct 30 18:54:11.437245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:11.437217    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:54:16.818110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.806183    5065 generic.go:356] "Generic (PLEG): container finished" podID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerID="4303b046109983061a4ad82eb713dc3f3989753a2322d0eaa0b41d9c698d99b1" exitCode=0
Oct 30 18:54:16.818110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.806304    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" event={"ID":"d383ad5a-1fa9-43f3-87aa-1b17ef345961","Type":"ContainerDied","Data":"4303b046109983061a4ad82eb713dc3f3989753a2322d0eaa0b41d9c698d99b1"}
Oct 30 18:54:16.836111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.822523    5065 generic.go:356] "Generic (PLEG): container finished" podID="f0ac827e-8d2a-482b-8596-27baeb496787" containerID="a130c48dfc6ce92efd37887f41d72052c581f8482ff6ef61da1154d9e9465bb6" exitCode=0
Oct 30 18:54:16.836111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.822653    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" event={"ID":"f0ac827e-8d2a-482b-8596-27baeb496787","Type":"ContainerDied","Data":"a130c48dfc6ce92efd37887f41d72052c581f8482ff6ef61da1154d9e9465bb6"}
Oct 30 18:54:16.836111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.825574    5065 generic.go:356] "Generic (PLEG): container finished" podID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerID="b5ab877112db8a14f71c669e38bc4e0650d238d605674ff5c1d50a00bad09f24" exitCode=0
Oct 30 18:54:16.836111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:16.825734    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" event={"ID":"b0bbebaa-6379-4191-a6b9-51f94eb9462d","Type":"ContainerDied","Data":"b5ab877112db8a14f71c669e38bc4e0650d238d605674ff5c1d50a00bad09f24"}
Oct 30 18:54:17.244243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.244215    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:54:17.276513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276486    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276545    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276571    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276588    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276620    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276671    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276785    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276806    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.276997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.276925    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\") pod \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\" (UID: \"b0bbebaa-6379-4191-a6b9-51f94eb9462d\") "
Oct 30 18:54:17.277801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.277772    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:17.278149 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.278059    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.278300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.278281    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.278586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.278565    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.278881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.278864    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.283388 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.279015    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.322240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.322041    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.322434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.322359    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.324295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.324267    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:17.344840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.344778    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474" pod "b0bbebaa-6379-4191-a6b9-51f94eb9462d" (UID: "b0bbebaa-6379-4191-a6b9-51f94eb9462d"). InnerVolumeSpecName "kube-api-access-8t474". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.347275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.346847    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:54:17.352365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.352345    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:54:17.379036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379018    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379047    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379117    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379151    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379178    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379219    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379256    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379300    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379331    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379358    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379386    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379517    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379615    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379648    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379697    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.379901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379805    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\") pod \"f0ac827e-8d2a-482b-8596-27baeb496787\" (UID: \"f0ac827e-8d2a-482b-8596-27baeb496787\") "
Oct 30 18:54:17.379901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379865    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.380183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.379905    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\") pod \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\" (UID: \"d383ad5a-1fa9-43f3-87aa-1b17ef345961\") "
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380651    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380675    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380692    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380706    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380720    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380733    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380746    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.380761    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t474\" (UniqueName: \"kubernetes.io/projected/b0bbebaa-6379-4191-a6b9-51f94eb9462d-kube-api-access-8t474\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.389163    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.398944    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.399216    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.399419    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.399597    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.399891    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.401559    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.401734    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.402297    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.402724    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.403386    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:17.403629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.403396    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.405144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.404496    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "kube-api-access-cq9bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.405292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.405264    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:17.405389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.405371    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.405777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.405750    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:17.405975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.405952    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t" pod "d383ad5a-1fa9-43f3-87aa-1b17ef345961" (UID: "d383ad5a-1fa9-43f3-87aa-1b17ef345961"). InnerVolumeSpecName "kube-api-access-mb48t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.406191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.406170    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token" pod "f0ac827e-8d2a-482b-8596-27baeb496787" (UID: "f0ac827e-8d2a-482b-8596-27baeb496787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.482402 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482368    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482402 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482398    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482413    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac827e-8d2a-482b-8596-27baeb496787-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482422    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482432    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482442    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482451    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482459    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb48t\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-kube-api-access-mb48t\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482468    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac827e-8d2a-482b-8596-27baeb496787-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482477    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482491    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482500    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac827e-8d2a-482b-8596-27baeb496787-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482510    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482518    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482526    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482535    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9bv\" (UniqueName: \"kubernetes.io/projected/f0ac827e-8d2a-482b-8596-27baeb496787-kube-api-access-cq9bv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482543    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.482606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.482551    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d383ad5a-1fa9-43f3-87aa-1b17ef345961-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.712870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.712142    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713541    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy"
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713566    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy"
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713585    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy"
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713593    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy"
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713610    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy"
Oct 30 18:54:17.730852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.713617    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy"
Oct 30 18:54:17.769300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.731616    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" containerName="istio-proxy"
Oct 30 18:54:17.769300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.731650    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" containerName="istio-proxy"
Oct 30 18:54:17.769300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.731664    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" containerName="istio-proxy"
Oct 30 18:54:17.780162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.778987    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:54:17.780162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.779200    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.788157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.788131    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fb19ff-openshift-default-dockercfg-bhqdc\""
Oct 30 18:54:17.812375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812330    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812385    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812516    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812587    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812629    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812656    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812765    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812841    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.812939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.812882    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh667\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.910619 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.910588    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"
Oct 30 18:54:17.970969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.970855    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.970969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.970954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.970992    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh667\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971048    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971235 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971077    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971306    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971388    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971435    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971462    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.971980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.971938    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.974280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.973946    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.974280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.974226    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.974456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.974310    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.979599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.979576    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:17.999131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:17.998507    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"
Oct 30 18:54:18.011029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.010300    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"
Oct 30 18:54:18.018440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.012958    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h" event={"ID":"d383ad5a-1fa9-43f3-87aa-1b17ef345961","Type":"ContainerDied","Data":"d5ac65a208c6f33a0820f25054174956efaefca40fb0f43b2a4ba1cb59e5f88f"}
Oct 30 18:54:18.018440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.012991    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n" event={"ID":"f0ac827e-8d2a-482b-8596-27baeb496787","Type":"ContainerDied","Data":"b3a667b72c75b8b4ab821add9d46da30720c8e51b06468f68ec502ba40243a50"}
Oct 30 18:54:18.018440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.013026    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf" event={"ID":"b0bbebaa-6379-4191-a6b9-51f94eb9462d","Type":"ContainerDied","Data":"c93121aaecd72d115947742bbeabdb577b49cd1f7b74117a2290ae14060583a3"}
Oct 30 18:54:18.018440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.013138    5065 scope.go:122] "RemoveContainer" containerID="4303b046109983061a4ad82eb713dc3f3989753a2322d0eaa0b41d9c698d99b1"
Oct 30 18:54:18.018440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.017306    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:18.020121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.019292    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:18.020121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.020017    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh667\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:18.020827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.020316    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\") pod \"e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") " pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:18.053115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.050071    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:54:18.056363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.056338    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-wsf7h"]
Oct 30 18:54:18.059079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.059059    5065 scope.go:122] "RemoveContainer" containerID="a130c48dfc6ce92efd37887f41d72052c581f8482ff6ef61da1154d9e9465bb6"
Oct 30 18:54:18.070720 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.070702    5065 scope.go:122] "RemoveContainer" containerID="b5ab877112db8a14f71c669e38bc4e0650d238d605674ff5c1d50a00bad09f24"
Oct 30 18:54:18.074866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.074837    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:54:18.084563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.084542    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nzvcf"]
Oct 30 18:54:18.137974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.137939    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:54:18.144871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.144844    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ntv8n"]
Oct 30 18:54:18.155542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.155515    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:18.466061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.465785    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:54:18.468677 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:54:18.468650    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b3cfc3_3416_4ac7_b939_22f839ad5639.slice/crio-7d1408e9bf3a03d1ba4d4bdc3eb2fc2a1bf8be2e5be2611ab04433cb66e2b407 WatchSource:0}: Error finding container 7d1408e9bf3a03d1ba4d4bdc3eb2fc2a1bf8be2e5be2611ab04433cb66e2b407: Status 404 returned error can't find the container with id 7d1408e9bf3a03d1ba4d4bdc3eb2fc2a1bf8be2e5be2611ab04433cb66e2b407
Oct 30 18:54:18.471773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.471733    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:18.471867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.471822    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:18.471907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:18.471868    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:19.024462 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.024424    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerStarted","Data":"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"}
Oct 30 18:54:19.024905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.024493    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerStarted","Data":"7d1408e9bf3a03d1ba4d4bdc3eb2fc2a1bf8be2e5be2611ab04433cb66e2b407"}
Oct 30 18:54:19.073115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.072473    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podStartSLOduration=2.072450559 podStartE2EDuration="2.072450559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:19.061497543 +0000 UTC m=+1279.324962526" watchObservedRunningTime="2026-10-30 18:54:19.072450559 +0000 UTC m=+1279.335915517"
Oct 30 18:54:19.160260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.160228    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:19.311212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.311029    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:19.650579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.650502    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"]
Oct 30 18:54:19.655198 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.655178    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.663201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.662624    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:19.663201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.662749    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:54:19.666317 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.666291    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"]
Oct 30 18:54:19.721828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.721783    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.722029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.721978    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.722140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.722061    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pc8\" (UniqueName: \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.823770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.823666    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.823984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.823952    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.824141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.824118    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pc8\" (UniqueName: \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.824793 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.824765    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.826775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.826750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.847131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.845498    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pc8\" (UniqueName: \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\") pod \"maas-api-revoke-sub-f77071f8-pp657\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:19.897711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.897672    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bbebaa-6379-4191-a6b9-51f94eb9462d" path="/var/lib/kubelet/pods/b0bbebaa-6379-4191-a6b9-51f94eb9462d/volumes"
Oct 30 18:54:19.898621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.898521    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d383ad5a-1fa9-43f3-87aa-1b17ef345961" path="/var/lib/kubelet/pods/d383ad5a-1fa9-43f3-87aa-1b17ef345961/volumes"
Oct 30 18:54:19.899217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.899193    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ac827e-8d2a-482b-8596-27baeb496787" path="/var/lib/kubelet/pods/f0ac827e-8d2a-482b-8596-27baeb496787/volumes"
Oct 30 18:54:19.998061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:19.997983    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:20.040581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:20.040523    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:20.042188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:20.041874    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:54:20.160281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:20.160240    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:54:20.160473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:20.160323    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:54:20.339537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:20.339504    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"]
Oct 30 18:54:20.340337 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:54:20.340302    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88bab4_0155_4aab_89c3_af0aa8b170c8.slice/crio-d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed WatchSource:0}: Error finding container d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed: Status 404 returned error can't find the container with id d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed
Oct 30 18:54:21.060917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:21.060119    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657" event={"ID":"8c88bab4-0155-4aab-89c3-af0aa8b170c8","Type":"ContainerStarted","Data":"15f4937d6db41568b5403243b2b4e5b620eda510c4962363dabf7b26334604be"}
Oct 30 18:54:21.060917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:21.060196    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657" event={"ID":"8c88bab4-0155-4aab-89c3-af0aa8b170c8","Type":"ContainerStarted","Data":"d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed"}
Oct 30 18:54:21.092900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:21.092831    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657" podStartSLOduration=2.092811939 podStartE2EDuration="2.092811939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:21.087060511 +0000 UTC m=+1281.350525459" watchObservedRunningTime="2026-10-30 18:54:21.092811939 +0000 UTC m=+1281.356276888"
Oct 30 18:54:22.077474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.077439    5065 generic.go:356] "Generic (PLEG): container finished" podID="8c88bab4-0155-4aab-89c3-af0aa8b170c8" containerID="15f4937d6db41568b5403243b2b4e5b620eda510c4962363dabf7b26334604be" exitCode=0
Oct 30 18:54:22.077988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.077540    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657" event={"ID":"8c88bab4-0155-4aab-89c3-af0aa8b170c8","Type":"ContainerDied","Data":"15f4937d6db41568b5403243b2b4e5b620eda510c4962363dabf7b26334604be"}
Oct 30 18:54:22.741518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.741418    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:54:22.746463 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.746391    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:54:22.746626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.746525    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.766548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.766509    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-59f140-openshift-default-dockercfg-f8fv6\""
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991526    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991585    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991643    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991676    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991725    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991757    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.991987    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.992024    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2npn\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:22.992161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:22.992128    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121849    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121891    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121966    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121999    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.122058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.122112    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.122372    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.122413    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2npn\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.122861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.121659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.124113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.123453    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.124192 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.124130    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.135160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.124659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.135160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.129433    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.146398 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:54:23.146358    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:54:23.149731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.148562    5065 generic.go:356] "Generic (PLEG): container finished" podID="79044e03-a475-4595-b8f3-04e6866d9558" containerID="3f1cf3ab5cdaf3d95f421d0a6e322f4b2235656a08bfdee41300a637226b329a" exitCode=0
Oct 30 18:54:23.149731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.148770    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerDied","Data":"3f1cf3ab5cdaf3d95f421d0a6e322f4b2235656a08bfdee41300a637226b329a"}
Oct 30 18:54:23.149731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.149284    5065 scope.go:122] "RemoveContainer" containerID="3f1cf3ab5cdaf3d95f421d0a6e322f4b2235656a08bfdee41300a637226b329a"
Oct 30 18:54:23.153926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.153743    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.153926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.153785    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2npn\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.154285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.154187    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.154394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.154288    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.154394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.154347    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.154719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.154655    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.157109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.156073    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\") pod \"e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") " pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.374631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.374590    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:23.398807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.398781    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:23.477294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.477253    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:54:23.478542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.478405    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8c88bab4-0155-4aab-89c3-af0aa8b170c8" containerName="revoke-keys"
Oct 30 18:54:23.478542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.478429    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c88bab4-0155-4aab-89c3-af0aa8b170c8" containerName="revoke-keys"
Oct 30 18:54:23.478711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.478598    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c88bab4-0155-4aab-89c3-af0aa8b170c8" containerName="revoke-keys"
Oct 30 18:54:23.494970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.494940    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:54:23.496225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.496199    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.528011    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fb19ff\""
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.528438    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fb19ff-dockercfg-nrn4f\""
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.529169    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\") pod \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") "
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.529205    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\") pod \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") "
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.529434    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\" (UniqueName: \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\") pod \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\" (UID: \"8c88bab4-0155-4aab-89c3-af0aa8b170c8\") "
Oct 30 18:54:23.535109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.532193    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca" pod "8c88bab4-0155-4aab-89c3-af0aa8b170c8" (UID: "8c88bab4-0155-4aab-89c3-af0aa8b170c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:23.541650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.538889    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:54:23.541650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.539000    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.547966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.547933    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:54:23.562813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.562549    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8" pod "8c88bab4-0155-4aab-89c3-af0aa8b170c8" (UID: "8c88bab4-0155-4aab-89c3-af0aa8b170c8"). InnerVolumeSpecName "kube-api-access-g4pc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.564580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.564033    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca" pod "8c88bab4-0155-4aab-89c3-af0aa8b170c8" (UID: "8c88bab4-0155-4aab-89c3-af0aa8b170c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:23.631722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.631593    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdhz\" (UniqueName: \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.631722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.631669    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsx7\" (UniqueName: \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.631953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.631857    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.631953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.631938    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.632214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.632191    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.632325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.632219    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c88bab4-0155-4aab-89c3-af0aa8b170c8-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.632325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.632236    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4pc8\" (UniqueName: \"kubernetes.io/projected/8c88bab4-0155-4aab-89c3-af0aa8b170c8-kube-api-access-g4pc8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.742111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.740170    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdhz\" (UniqueName: \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.742111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.740252    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsx7\" (UniqueName: \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.742111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.740367    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.742111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.740428    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.805210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.769291    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:23.828405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.817636    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.828405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.817724    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.828405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.817777    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:23.861249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.849790    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsx7\" (UniqueName: \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.861249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.858722    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.861249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.860625    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:23.881323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.871683    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:54:23.991110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:23.990112    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdhz\" (UniqueName: \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\") pod \"payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:24.122163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.121391    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:24.180764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.178135    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657" event={"ID":"8c88bab4-0155-4aab-89c3-af0aa8b170c8","Type":"ContainerDied","Data":"d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed"}
Oct 30 18:54:24.180764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.178197    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71ce7bb6b263ce540a70d5f95ddad190821014fb7f029df2e0be82168be86ed"
Oct 30 18:54:24.180764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.178274    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"
Oct 30 18:54:24.185207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.183811    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerStarted","Data":"806d216febacd8d6b5ab288f5ef3dbd5fc325145fcf77b120f7bf2d1caa1fce9"}
Oct 30 18:54:24.186946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.186764    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerStarted","Data":"b5ba0eab09260aebf29b674aa704ca04c09368128d1f23106ab5d41bca439203"}
Oct 30 18:54:24.663699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.663251    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:54:24.696210 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.693123    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:54:24.913184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.891794    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:54:24.913184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.891918    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:24.913184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:24.891960    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:25.335131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.334630    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerStarted","Data":"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"}
Oct 30 18:54:25.340987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.340942    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" event={"ID":"e26eb598-5229-438c-ab5e-63a432dafd03","Type":"ContainerStarted","Data":"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"}
Oct 30 18:54:25.341184 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.341026    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" event={"ID":"e26eb598-5229-438c-ab5e-63a432dafd03","Type":"ContainerStarted","Data":"26115948ceeced9434005e9bbf2f8a0640e67beb6dc0a0d6b62eba6fe6050c3f"}
Oct 30 18:54:25.341855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.341833    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:25.344667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.344639    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" event={"ID":"c9153812-81d3-44e7-90b0-4e2c4922c240","Type":"ContainerStarted","Data":"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"}
Oct 30 18:54:25.344797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.344694    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" event={"ID":"c9153812-81d3-44e7-90b0-4e2c4922c240","Type":"ContainerStarted","Data":"5187e51f4e61075928292b2b3ca2266b1c1b9d99fed827097a7202d134eca58e"}
Oct 30 18:54:25.344911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.344892    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:25.375684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.375663    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:25.385401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.385377    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:25.427104 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.426711    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podStartSLOduration=3.426690934 podStartE2EDuration="3.426690934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.378307965 +0000 UTC m=+1285.641772918" watchObservedRunningTime="2026-10-30 18:54:25.426690934 +0000 UTC m=+1285.690155882"
Oct 30 18:54:25.428946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.428177    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" podStartSLOduration=2.42816424 podStartE2EDuration="2.42816424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.40847235 +0000 UTC m=+1285.671937299" watchObservedRunningTime="2026-10-30 18:54:25.42816424 +0000 UTC m=+1285.691629187"
Oct 30 18:54:25.446750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.446705    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" podStartSLOduration=2.4466935530000002 podStartE2EDuration="2.446693553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.444879288 +0000 UTC m=+1285.708344237" watchObservedRunningTime="2026-10-30 18:54:25.446693553 +0000 UTC m=+1285.710158501"
Oct 30 18:54:25.894682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.894651    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:25.896008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:25.895979    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:26.354632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:26.354560    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:26.355966 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:26.355939    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:54:26.907111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:26.900184    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:26.907111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:26.900245    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:27.888203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:27.888166    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:27.888784 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:27.888751    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:28.886117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:28.883415    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:28.886117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:28.883470    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:29.911110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:29.900657    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 18:54:29.911110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:29.900714    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 18:54:30.310664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.310620    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:54:30.316745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.316716    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.325740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.325700    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:54:30.328888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.328860    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9d42ff\""
Oct 30 18:54:30.329145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.329128    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9d42ff-dockercfg-pnn8d\""
Oct 30 18:54:30.365475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.365437    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:54:30.371207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.371186    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.406987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.405333    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:54:30.675162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.674424    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dql\" (UniqueName: \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.675162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.674534    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.801844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.801812    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.802035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.801957    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbm4\" (UniqueName: \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.802035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.802020    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.802296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.802267    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dql\" (UniqueName: \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.803370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.803345    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.815947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.815925    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dql\" (UniqueName: \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\") pod \"payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:30.885401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.885376    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:54:30.886151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.886129    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:54:30.886771 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.886754    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:54:30.904446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.904422    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbm4\" (UniqueName: \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.904566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.904469    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.905345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.905323    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.927219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.927156    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbm4\" (UniqueName: \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\") pod \"payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:30.953271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:30.953247    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:31.013802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.013675    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:31.501925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.501576    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:54:31.502235 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:54:31.502174    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e2f9f2_810f_4dac_b56d_35d0618b17fe.slice/crio-7b4c859775057410bc675306cc390d966d4c6e3bdeb1a0c2dd7435d0d8c53ecb WatchSource:0}: Error finding container 7b4c859775057410bc675306cc390d966d4c6e3bdeb1a0c2dd7435d0d8c53ecb: Status 404 returned error can't find the container with id 7b4c859775057410bc675306cc390d966d4c6e3bdeb1a0c2dd7435d0d8c53ecb
Oct 30 18:54:31.600137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.599257    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:54:31.687042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.686971    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" event={"ID":"fc104eac-5018-4ba7-b042-b7fd075f1fdb","Type":"ContainerStarted","Data":"296eec42519e9f3ff33f280b0af5f5872dad474ce0da2fc65e7d2bbcae6c92af"}
Oct 30 18:54:31.690649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.689949    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" event={"ID":"13e2f9f2-810f-4dac-b56d-35d0618b17fe","Type":"ContainerStarted","Data":"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"}
Oct 30 18:54:31.690649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.689992    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" event={"ID":"13e2f9f2-810f-4dac-b56d-35d0618b17fe","Type":"ContainerStarted","Data":"7b4c859775057410bc675306cc390d966d4c6e3bdeb1a0c2dd7435d0d8c53ecb"}
Oct 30 18:54:31.690649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.690031    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:31.719539 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:31.719475    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" podStartSLOduration=1.719457297 podStartE2EDuration="1.719457297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:31.714795345 +0000 UTC m=+1291.978260293" watchObservedRunningTime="2026-10-30 18:54:31.719457297 +0000 UTC m=+1291.982922246"
Oct 30 18:54:32.702978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:32.702939    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" event={"ID":"fc104eac-5018-4ba7-b042-b7fd075f1fdb","Type":"ContainerStarted","Data":"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"}
Oct 30 18:54:32.703460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:32.703016    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:32.723831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:32.723774    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" podStartSLOduration=2.723756924 podStartE2EDuration="2.723756924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:32.722381357 +0000 UTC m=+1292.985846325" watchObservedRunningTime="2026-10-30 18:54:32.723756924 +0000 UTC m=+1292.987221872"
Oct 30 18:54:36.358246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:36.358215    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:54:37.365786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:37.365751    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:54:42.716181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:42.715708    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:54:43.730849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:43.730820    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:54:48.982533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.982502    5065 generic.go:356] "Generic (PLEG): container finished" podID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerID="d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909" exitCode=0
Oct 30 18:54:48.983026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.982582    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerDied","Data":"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"}
Oct 30 18:54:48.983073 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.983062    5065 scope.go:122] "RemoveContainer" containerID="d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"
Oct 30 18:54:48.984907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.984879    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:48.984985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.984955    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:48.985028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:48.984986    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:54:50.014871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:50.014840    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerStarted","Data":"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"}
Oct 30 18:54:50.977273 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:50.977245    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:54:50.978764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:50.978739    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 18:54:50.978906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:50.978785    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 18:54:51.977954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:51.977916    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 18:54:51.978406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:51.977972    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 18:54:52.977490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:52.977449    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 18:54:52.977682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:52.977526    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 18:54:53.057548 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:54:53.057514    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:54:53.977563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:53.977532    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 18:54:53.977731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:53.977589    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 18:54:54.335031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:54.335002    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:54:54.977079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:54.977041    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 18:54:54.977079 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:54.977110    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 18:54:55.982152 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:55.982076    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:54:55.983145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:54:55.983079    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:55:00.833575 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.833528    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"]
Oct 30 18:55:00.841002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.840032    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:00.848505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.848424    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:00.850597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.850570    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:55:00.853010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.852974    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"]
Oct 30 18:55:00.968337 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.968303    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:00.968542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.968485    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:00.968542 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:00.968536    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9769h\" (UniqueName: \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.071122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.069881    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.071122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.069958    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9769h\" (UniqueName: \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.071122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.070238    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.115175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.099323    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9769h\" (UniqueName: \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.115175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.106490    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.132796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.131659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.183158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.181545    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:01.584197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:01.584158    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"]
Oct 30 18:55:01.591048 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:01.591000    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb582f584_ee85_40ab_b941_0a06d429ae35.slice/crio-44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168 WatchSource:0}: Error finding container 44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168: Status 404 returned error can't find the container with id 44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168
Oct 30 18:55:02.038196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.038149    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"]
Oct 30 18:55:02.073286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.072039    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.086120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.075928    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"]
Oct 30 18:55:02.301340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.289374    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.301340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.289457    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.301340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.289518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkxs\" (UniqueName: \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.336767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.336685    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp" event={"ID":"b582f584-ee85-40ab-b941-0a06d429ae35","Type":"ContainerStarted","Data":"44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168"}
Oct 30 18:55:02.436113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.422641    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.436113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.422739    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.436113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.422814    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkxs\" (UniqueName: \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.436113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.425103    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.581114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.577947    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.581114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.578625    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkxs\" (UniqueName: \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.645121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.644362    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:02.970180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.969939    5065 scope.go:122] "RemoveContainer" containerID="efe4cab9bc8311499220fbe0a9bf315953fc283a770cf9fa623069c0352c6649"
Oct 30 18:55:03.007898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:02.989029    5065 scope.go:122] "RemoveContainer" containerID="27670cdbafd10eea32ad8e1ea6a5033987409029dcbd9fdc4da4790b6efbbc4c"
Oct 30 18:55:03.343646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:03.304853    5065 scope.go:122] "RemoveContainer" containerID="8d31e8d7b5df40aa2cb9e42a4b4715cd7b89ff7d7f65983d720983cb91bc7da2"
Oct 30 18:55:03.546294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:03.544965    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"]
Oct 30 18:55:03.579115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:03.570458    5065 scope.go:122] "RemoveContainer" containerID="a264e1ce4de38840c63a3d65bc89d8ceb42ec6ff00ff8a177c6578958ba59b56"
Oct 30 18:55:03.997844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:03.997815    5065 scope.go:122] "RemoveContainer" containerID="04f94974e194d2f3049395a7cfe72c02e5d35d8b23a19dd81706438502bd97c0"
Oct 30 18:55:04.326113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.305607    5065 generic.go:356] "Generic (PLEG): container finished" podID="b582f584-ee85-40ab-b941-0a06d429ae35" containerID="a74acb8b0f6061bd89c311f001dc2bfb78ff02f7bbbbe2c9e4842cc87784eaac" exitCode=0
Oct 30 18:55:04.369112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.349005    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp" event={"ID":"b582f584-ee85-40ab-b941-0a06d429ae35","Type":"ContainerDied","Data":"a74acb8b0f6061bd89c311f001dc2bfb78ff02f7bbbbe2c9e4842cc87784eaac"}
Oct 30 18:55:04.389496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.373307    5065 scope.go:122] "RemoveContainer" containerID="25b7f4bf27f7d268445a0809d02a16de8224161ec9b70059e5bf7f1e51053e22"
Oct 30 18:55:04.701241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.701195    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:55:04.760327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.760164    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:04.770943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.770902    5065 scope.go:122] "RemoveContainer" containerID="916c1391308e3db925613bed99a67d39995268ce5d4844dfb54e297aad424a9c"
Oct 30 18:55:04.836812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:04.836762    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:55:05.012906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.012310    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.054487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.054336    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:55:05.126114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.112297    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.157560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.157533    5065 scope.go:122] "RemoveContainer" containerID="5a9db873f00fcca123e110f3aad4d20458b1e1bd6ecc2d83124278a0b27a082e"
Oct 30 18:55:05.409120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.406145    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg" event={"ID":"8eec2306-98cc-495b-b705-86026beda714","Type":"ContainerStarted","Data":"877d35f3e4eb0a2642cd18cf8047fb6251d342551b5a08ff08e8ee624fcd74cc"}
Oct 30 18:55:05.410847 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.410754    5065 scope.go:122] "RemoveContainer" containerID="c3388f291222306f9b5096b35352d58c2850c3703d5f6c21b33da776aa93dfb0"
Oct 30 18:55:05.705406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.705374    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.727419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.727390    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:55:05.731850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.731817    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.754286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.754238    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.760835 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.760805    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:55:05.775915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.772589    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.775915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.772925    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:55:05.851563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.851530    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:05.861130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:05.855540    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:55:06.028716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.028177    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:06.075500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.061386    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:06.667196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.642825    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:55:06.667196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.642887    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:55:06.667196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.642932    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:55:06.759114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.754959    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:06.789905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.789866    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:55:06.811039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.811002    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:06.840239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.840148    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg" event={"ID":"8eec2306-98cc-495b-b705-86026beda714","Type":"ContainerStarted","Data":"0880ecae894aeec6e162d65f80b5466cb3f6ddf2ddd9404005b4c842b9ca6558"}
Oct 30 18:55:06.840522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.840506    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:55:06.840661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.840650    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:55:06.869144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:06.869102    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.123254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.123166    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.157128 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.153629    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:55:07.586631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.586587    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:55:07.613785 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.593404    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:55:07.794341 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.762253    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.810126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.804223    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:55:07.813658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.813618    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.874113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.833361    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.876642 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.876608    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:07.971067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:07.959900    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:08.034119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.004847    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\") pod \"b582f584-ee85-40ab-b941-0a06d429ae35\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") "
Oct 30 18:55:08.034119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.004923    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\") pod \"b582f584-ee85-40ab-b941-0a06d429ae35\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") "
Oct 30 18:55:08.034119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.004997    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\" (UniqueName: \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\") pod \"b582f584-ee85-40ab-b941-0a06d429ae35\" (UID: \"b582f584-ee85-40ab-b941-0a06d429ae35\") "
Oct 30 18:55:08.034119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.010562    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca" pod "b582f584-ee85-40ab-b941-0a06d429ae35" (UID: "b582f584-ee85-40ab-b941-0a06d429ae35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:08.034119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.023278    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.064131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.064068    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h" pod "b582f584-ee85-40ab-b941-0a06d429ae35" (UID: "b582f584-ee85-40ab-b941-0a06d429ae35"). InnerVolumeSpecName "kube-api-access-9769h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:08.137312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.088363    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca" pod "b582f584-ee85-40ab-b941-0a06d429ae35" (UID: "b582f584-ee85-40ab-b941-0a06d429ae35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:08.203113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.171967    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b582f584-ee85-40ab-b941-0a06d429ae35-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.203113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.172008    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9769h\" (UniqueName: \"kubernetes.io/projected/b582f584-ee85-40ab-b941-0a06d429ae35-kube-api-access-9769h\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.203113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.172078    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:55:08.234727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.234608    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg" podStartSLOduration=7.23458957 podStartE2EDuration="7.23458957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:07.558490935 +0000 UTC m=+1327.821955884" watchObservedRunningTime="2026-10-30 18:55:08.23458957 +0000 UTC m=+1328.498054520"
Oct 30 18:55:08.425261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.424123    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:55:08.425261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.424212    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:55:08.425261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.424251    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:55:08.425261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.424285    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:55:08.498166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.490361    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:55:08.563398 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.563368    5065 generic.go:356] "Generic (PLEG): container finished" podID="8eec2306-98cc-495b-b705-86026beda714" containerID="0880ecae894aeec6e162d65f80b5466cb3f6ddf2ddd9404005b4c842b9ca6558" exitCode=0
Oct 30 18:55:08.563805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.563476    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg" event={"ID":"8eec2306-98cc-495b-b705-86026beda714","Type":"ContainerDied","Data":"0880ecae894aeec6e162d65f80b5466cb3f6ddf2ddd9404005b4c842b9ca6558"}
Oct 30 18:55:08.567246 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.567223    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"
Oct 30 18:55:08.570948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.568150    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp" event={"ID":"b582f584-ee85-40ab-b941-0a06d429ae35","Type":"ContainerDied","Data":"44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168"}
Oct 30 18:55:08.570948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:08.568197    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44ed5ab54cad52b2c377e6e1dceae5a8117ecbd1e8a8f6d7abada5aa3da1b168"
Oct 30 18:55:10.050354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.050323    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:10.155267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.155234    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\" (UniqueName: \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\") pod \"8eec2306-98cc-495b-b705-86026beda714\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") "
Oct 30 18:55:10.155465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.155302    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\") pod \"8eec2306-98cc-495b-b705-86026beda714\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") "
Oct 30 18:55:10.155465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.155394    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\") pod \"8eec2306-98cc-495b-b705-86026beda714\" (UID: \"8eec2306-98cc-495b-b705-86026beda714\") "
Oct 30 18:55:10.156680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.156658    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca" pod "8eec2306-98cc-495b-b705-86026beda714" (UID: "8eec2306-98cc-495b-b705-86026beda714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:10.159169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.159147    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca" pod "8eec2306-98cc-495b-b705-86026beda714" (UID: "8eec2306-98cc-495b-b705-86026beda714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:10.159674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.159647    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs" pod "8eec2306-98cc-495b-b705-86026beda714" (UID: "8eec2306-98cc-495b-b705-86026beda714"). InnerVolumeSpecName "kube-api-access-txkxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:10.257238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.257149    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkxs\" (UniqueName: \"kubernetes.io/projected/8eec2306-98cc-495b-b705-86026beda714-kube-api-access-txkxs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:10.257238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.257181    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eec2306-98cc-495b-b705-86026beda714-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:10.257238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.257190    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eec2306-98cc-495b-b705-86026beda714-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:10.626725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.626698    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"
Oct 30 18:55:10.627902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.627867    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg" event={"ID":"8eec2306-98cc-495b-b705-86026beda714","Type":"ContainerDied","Data":"877d35f3e4eb0a2642cd18cf8047fb6251d342551b5a08ff08e8ee624fcd74cc"}
Oct 30 18:55:10.628052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:10.627932    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877d35f3e4eb0a2642cd18cf8047fb6251d342551b5a08ff08e8ee624fcd74cc"
Oct 30 18:55:11.015943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:11.015911    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:55:13.154513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:13.154011    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"]
Oct 30 18:55:13.161222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:13.161190    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5ab51-58bfp"]
Oct 30 18:55:13.896585 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:13.896542    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b582f584-ee85-40ab-b941-0a06d429ae35" path="/var/lib/kubelet/pods/b582f584-ee85-40ab-b941-0a06d429ae35/volumes"
Oct 30 18:55:16.016818 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.016751    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" podUID="0f67c87c-2473-4870-9d0a-3039d136b7ae" containerName="payload-pre-processing" containerID="cri-o://77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c" gracePeriod=25
Oct 30 18:55:16.451742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.451716    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:55:16.536779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.536752    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\") pod \"0f67c87c-2473-4870-9d0a-3039d136b7ae\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") "
Oct 30 18:55:16.536955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.536934    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\" (UniqueName: \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\") pod \"0f67c87c-2473-4870-9d0a-3039d136b7ae\" (UID: \"0f67c87c-2473-4870-9d0a-3039d136b7ae\") "
Oct 30 18:55:16.541060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.541032    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn" pod "0f67c87c-2473-4870-9d0a-3039d136b7ae" (UID: "0f67c87c-2473-4870-9d0a-3039d136b7ae"). InnerVolumeSpecName "kube-api-access-hcgsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:16.560959 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.560931    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume" pod "0f67c87c-2473-4870-9d0a-3039d136b7ae" (UID: "0f67c87c-2473-4870-9d0a-3039d136b7ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:16.643683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.641921    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f67c87c-2473-4870-9d0a-3039d136b7ae-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.643683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.641961    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgsn\" (UniqueName: \"kubernetes.io/projected/0f67c87c-2473-4870-9d0a-3039d136b7ae-kube-api-access-hcgsn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.872513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.872292    5065 generic.go:356] "Generic (PLEG): container finished" podID="0f67c87c-2473-4870-9d0a-3039d136b7ae" containerID="77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c" exitCode=0
Oct 30 18:55:16.872513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.872332    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" event={"ID":"0f67c87c-2473-4870-9d0a-3039d136b7ae","Type":"ContainerDied","Data":"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"}
Oct 30 18:55:16.872513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.872386    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t" event={"ID":"0f67c87c-2473-4870-9d0a-3039d136b7ae","Type":"ContainerDied","Data":"7a00489fe1ce8f7fd9b73ff8038625c4673dcd0e4016cf3d34b3b3e049b9a121"}
Oct 30 18:55:16.872513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.872342    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"
Oct 30 18:55:16.872513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.872409    5065 scope.go:122] "RemoveContainer" containerID="77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"
Oct 30 18:55:16.908868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.905170    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:55:16.910030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.910011    5065 scope.go:122] "RemoveContainer" containerID="77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"
Oct 30 18:55:16.910463 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:16.910437    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c\": container with ID starting with 77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c not found: ID does not exist" containerID="77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"
Oct 30 18:55:16.910566 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.910477    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c"} err="failed to get container status \"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c\": rpc error: code = NotFound desc = could not find container \"77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c\": container with ID starting with 77037c3c118cb519324bc777484d9f9ee451b789bd0cfbf78a66040df7cb2b4c not found: ID does not exist"
Oct 30 18:55:16.911272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:16.911252    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5ab51-5588f7dc7f-jzg5t"]
Oct 30 18:55:17.887385 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.887349    5065 generic.go:356] "Generic (PLEG): container finished" podID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerID="794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359" exitCode=0
Oct 30 18:55:17.889738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.889707    5065 generic.go:356] "Generic (PLEG): container finished" podID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerID="c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2" exitCode=0
Oct 30 18:55:17.894407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.894357    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f67c87c-2473-4870-9d0a-3039d136b7ae" path="/var/lib/kubelet/pods/0f67c87c-2473-4870-9d0a-3039d136b7ae/volumes"
Oct 30 18:55:17.894857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.894834    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerDied","Data":"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"}
Oct 30 18:55:17.894929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.894899    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerDied","Data":"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"}
Oct 30 18:55:17.895833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.895395    5065 scope.go:122] "RemoveContainer" containerID="c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"
Oct 30 18:55:17.895833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.895592    5065 scope.go:122] "RemoveContainer" containerID="794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"
Oct 30 18:55:17.897436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.897403    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:17.897525 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.897489    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:17.897568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.897537    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:17.912476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.912448    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:17.912594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.912523    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:17.912594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:17.912556    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:55:18.971148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:18.970862    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerStarted","Data":"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"}
Oct 30 18:55:18.978311 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:18.978284    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerStarted","Data":"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"}
Oct 30 18:55:19.156484 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.156383    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:55:19.157526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.157498    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:19.157662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.157554    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:19.390112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.389528    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:19.390112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.389677    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:19.390112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:19.389725    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:20.041903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:20.041876    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:55:20.155948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:20.155917    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:20.156238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:20.156210    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:20.376292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:20.376209    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:20.376510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:20.376488    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:21.169124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.161482    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:21.169124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.161542    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:21.532196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.501498    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:21.532196 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.501556    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:21.855229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.855139    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"]
Oct 30 18:55:21.856615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.856587    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b582f584-ee85-40ab-b941-0a06d429ae35" containerName="revoke-keys"
Oct 30 18:55:21.856787 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.856771    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b582f584-ee85-40ab-b941-0a06d429ae35" containerName="revoke-keys"
Oct 30 18:55:21.856934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.856921    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8eec2306-98cc-495b-b705-86026beda714" containerName="revoke-keys"
Oct 30 18:55:21.857114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857079    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eec2306-98cc-495b-b705-86026beda714" containerName="revoke-keys"
Oct 30 18:55:21.857236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857225    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0f67c87c-2473-4870-9d0a-3039d136b7ae" containerName="payload-pre-processing"
Oct 30 18:55:21.857310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857299    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f67c87c-2473-4870-9d0a-3039d136b7ae" containerName="payload-pre-processing"
Oct 30 18:55:21.857578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857568    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b582f584-ee85-40ab-b941-0a06d429ae35" containerName="revoke-keys"
Oct 30 18:55:21.857663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857654    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eec2306-98cc-495b-b705-86026beda714" containerName="revoke-keys"
Oct 30 18:55:21.857744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.857735    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f67c87c-2473-4870-9d0a-3039d136b7ae" containerName="payload-pre-processing"
Oct 30 18:55:21.867002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.866980    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"]
Oct 30 18:55:21.867293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.867263    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:21.873278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.873253    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:21.875240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.875218    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:55:21.968214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.968171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:21.968420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.968283    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkpx\" (UniqueName: \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:21.968504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:21.968425    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.070203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.070169    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.070382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.070301    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkpx\" (UniqueName: \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.070434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.070424    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.071132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.071061    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.072814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.072789    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.114158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.111794    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkpx\" (UniqueName: \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\") pod \"maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.156400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.156357    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:22.156570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.156424    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:22.200110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.193283    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:22.381175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.375915    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:22.381175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.375974    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:22.953780 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:22.951649    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"]
Oct 30 18:55:22.958782 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:22.958660    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fffd1f_7f16_4a13_85ab_6b78f2384220.slice/crio-35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c WatchSource:0}: Error finding container 35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c: Status 404 returned error can't find the container with id 35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c
Oct 30 18:55:23.075123 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:23.074371    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:55:23.173530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.173493    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:23.173755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.173561    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:23.176048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.176002    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr" event={"ID":"98fffd1f-7f16-4a13-85ab-6b78f2384220","Type":"ContainerStarted","Data":"35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c"}
Oct 30 18:55:23.417116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.407111    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:23.417116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.407169    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:23.741151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.732685    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"]
Oct 30 18:55:23.777696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.762433    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2ceb5-xtcxg"]
Oct 30 18:55:23.979177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:23.978247    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eec2306-98cc-495b-b705-86026beda714" path="/var/lib/kubelet/pods/8eec2306-98cc-495b-b705-86026beda714/volumes"
Oct 30 18:55:24.161513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.161467    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 18:55:24.161695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.161574    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 18:55:24.254755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.254656    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr" event={"ID":"98fffd1f-7f16-4a13-85ab-6b78f2384220","Type":"ContainerStarted","Data":"5ff658a6819810df2483686c51b263760fe618ad20b4a52502c435713fe40fca"}
Oct 30 18:55:24.339183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.331446    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr" podStartSLOduration=3.33142327 podStartE2EDuration="3.33142327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:24.295933519 +0000 UTC m=+1344.559398470" watchObservedRunningTime="2026-10-30 18:55:24.33142327 +0000 UTC m=+1344.594888220"
Oct 30 18:55:24.376692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.376658    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 18:55:24.395334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:24.386166    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 18:55:25.250275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.250230    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:55:25.253220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.253192    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:55:25.269346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.269317    5065 generic.go:356] "Generic (PLEG): container finished" podID="98fffd1f-7f16-4a13-85ab-6b78f2384220" containerID="5ff658a6819810df2483686c51b263760fe618ad20b4a52502c435713fe40fca" exitCode=0
Oct 30 18:55:25.269520 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.269400    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr" event={"ID":"98fffd1f-7f16-4a13-85ab-6b78f2384220","Type":"ContainerDied","Data":"5ff658a6819810df2483686c51b263760fe618ad20b4a52502c435713fe40fca"}
Oct 30 18:55:25.383111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.382724    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:25.385140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.383770    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:25.385140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:25.384754    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:26.460526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.460503    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:26.601488 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.601414    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\") pod \"98fffd1f-7f16-4a13-85ab-6b78f2384220\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") "
Oct 30 18:55:26.601651 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.601507    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\") pod \"98fffd1f-7f16-4a13-85ab-6b78f2384220\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") "
Oct 30 18:55:26.601740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.601704    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\" (UniqueName: \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\") pod \"98fffd1f-7f16-4a13-85ab-6b78f2384220\" (UID: \"98fffd1f-7f16-4a13-85ab-6b78f2384220\") "
Oct 30 18:55:26.601845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.601821    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca" pod "98fffd1f-7f16-4a13-85ab-6b78f2384220" (UID: "98fffd1f-7f16-4a13-85ab-6b78f2384220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:26.602226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.602203    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.604481 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.604453    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca" pod "98fffd1f-7f16-4a13-85ab-6b78f2384220" (UID: "98fffd1f-7f16-4a13-85ab-6b78f2384220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:26.604573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.604502    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx" pod "98fffd1f-7f16-4a13-85ab-6b78f2384220" (UID: "98fffd1f-7f16-4a13-85ab-6b78f2384220"). InnerVolumeSpecName "kube-api-access-mvkpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:26.703387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.703358    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkpx\" (UniqueName: \"kubernetes.io/projected/98fffd1f-7f16-4a13-85ab-6b78f2384220-kube-api-access-mvkpx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.703387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:26.703386    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fffd1f-7f16-4a13-85ab-6b78f2384220-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:27.291541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:27.291511    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"
Oct 30 18:55:27.291775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:27.291545    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr" event={"ID":"98fffd1f-7f16-4a13-85ab-6b78f2384220","Type":"ContainerDied","Data":"35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c"}
Oct 30 18:55:27.291775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:27.291617    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35be81255d906df5d8582f3b524304cdbaadc2500dfcbef05c80f9f0ed38211c"
Oct 30 18:55:31.066116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.066068    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:55:31.066495 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.066291    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy" containerID="cri-o://4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609" gracePeriod=30
Oct 30 18:55:31.073827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.073803    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"]
Oct 30 18:55:31.086028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.086005    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a4ad8-p8phl"]
Oct 30 18:55:31.559226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.559197    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"]
Oct 30 18:55:31.559933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.559917    5065 cpu_manager.go:414] "containerMap: removing container" podUID="98fffd1f-7f16-4a13-85ab-6b78f2384220" containerName="revoke-keys"
Oct 30 18:55:31.559933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.559933    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fffd1f-7f16-4a13-85ab-6b78f2384220" containerName="revoke-keys"
Oct 30 18:55:31.560108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.560097    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fffd1f-7f16-4a13-85ab-6b78f2384220" containerName="revoke-keys"
Oct 30 18:55:31.564185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.564167    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.581279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.581252    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"]
Oct 30 18:55:31.581876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.581846    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzdf\" (UniqueName: \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.581953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.581909    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.582154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.582131    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.592848 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.592819    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:31.593014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.592988    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:55:31.683176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.683148    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzdf\" (UniqueName: \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.683340 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.683236    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.683421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.683404    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.684251 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.684223    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.686397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.686377    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.697744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.697720    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzdf\" (UniqueName: \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\") pod \"maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.879903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.879819    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:31.901718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.901685    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175fb5b0-78d1-4d37-8940-795baba6a1b3" path="/var/lib/kubelet/pods/175fb5b0-78d1-4d37-8940-795baba6a1b3/volumes"
Oct 30 18:55:31.985643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.984310    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:55:31.999162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:31.998408    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:55:32.200560 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:32.200527    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52636fc_06de_4196_9599_75f8ff82fbbc.slice/crio-25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5 WatchSource:0}: Error finding container 25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5: Status 404 returned error can't find the container with id 25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5
Oct 30 18:55:32.200971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:32.200670    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"]
Oct 30 18:55:32.418125 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:32.415260    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4" event={"ID":"a52636fc-06de-4196-9599-75f8ff82fbbc","Type":"ContainerStarted","Data":"25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5"}
Oct 30 18:55:33.425938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:33.425849    5065 generic.go:356] "Generic (PLEG): container finished" podID="a52636fc-06de-4196-9599-75f8ff82fbbc" containerID="adc251a81bc890b62a933b7051fcb424404a49b6548a2217705c7df46748259b" exitCode=0
Oct 30 18:55:33.425938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:33.425925    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4" event={"ID":"a52636fc-06de-4196-9599-75f8ff82fbbc","Type":"ContainerDied","Data":"adc251a81bc890b62a933b7051fcb424404a49b6548a2217705c7df46748259b"}
Oct 30 18:55:34.153464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.152562    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"]
Oct 30 18:55:34.160199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.160168    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9d42ff-49qjr"]
Oct 30 18:55:34.615055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.615034    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:34.645071 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.645044    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\") pod \"a52636fc-06de-4196-9599-75f8ff82fbbc\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") "
Oct 30 18:55:34.645300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.645283    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\") pod \"a52636fc-06de-4196-9599-75f8ff82fbbc\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") "
Oct 30 18:55:34.645357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.645322    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\" (UniqueName: \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\") pod \"a52636fc-06de-4196-9599-75f8ff82fbbc\" (UID: \"a52636fc-06de-4196-9599-75f8ff82fbbc\") "
Oct 30 18:55:34.645662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.645637    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca" pod "a52636fc-06de-4196-9599-75f8ff82fbbc" (UID: "a52636fc-06de-4196-9599-75f8ff82fbbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:34.645865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.645849    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.647380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.647354    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca" pod "a52636fc-06de-4196-9599-75f8ff82fbbc" (UID: "a52636fc-06de-4196-9599-75f8ff82fbbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:34.647529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.647514    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf" pod "a52636fc-06de-4196-9599-75f8ff82fbbc" (UID: "a52636fc-06de-4196-9599-75f8ff82fbbc"). InnerVolumeSpecName "kube-api-access-mrzdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:34.747193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.747110    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzdf\" (UniqueName: \"kubernetes.io/projected/a52636fc-06de-4196-9599-75f8ff82fbbc-kube-api-access-mrzdf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.747193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:34.747147    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52636fc-06de-4196-9599-75f8ff82fbbc-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:35.446379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:35.446352    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"
Oct 30 18:55:35.446545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:35.446382    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4" event={"ID":"a52636fc-06de-4196-9599-75f8ff82fbbc","Type":"ContainerDied","Data":"25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5"}
Oct 30 18:55:35.446545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:35.446429    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25dcdbdbb16c8e6df8135c0c63c585758fa54a8471e2b1d4bd3ac5ebc6c16db5"
Oct 30 18:55:35.890989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:35.890958    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fffd1f-7f16-4a13-85ab-6b78f2384220" path="/var/lib/kubelet/pods/98fffd1f-7f16-4a13-85ab-6b78f2384220/volumes"
Oct 30 18:55:36.376449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.376414    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:55:36.458579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.458482    5065 generic.go:356] "Generic (PLEG): container finished" podID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerID="4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609" exitCode=0
Oct 30 18:55:36.458579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.458548    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"
Oct 30 18:55:36.458579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.458565    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerDied","Data":"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"}
Oct 30 18:55:36.458877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.458634    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx" event={"ID":"2fc997a6-a405-4237-8a5d-4170e806d0fa","Type":"ContainerDied","Data":"f2d89ec2601e1abf9731d0b099f8909b2a92fad44d2f2c7d7c5a26e330b4f94a"}
Oct 30 18:55:36.458877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.458667    5065 scope.go:122] "RemoveContainer" containerID="4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"
Oct 30 18:55:36.470632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470611    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.470761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470678    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.470761 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470751    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.470892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470853    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.470892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470876    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.471007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470907    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.471007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470943    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.471007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.470984    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.471228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.471012    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\") pod \"2fc997a6-a405-4237-8a5d-4170e806d0fa\" (UID: \"2fc997a6-a405-4237-8a5d-4170e806d0fa\") "
Oct 30 18:55:36.472258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.471005    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:36.472258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.471493    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:36.472258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.471900    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:36.472258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.472049    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:36.472258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.472136    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:36.474155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.474129    5065 scope.go:122] "RemoveContainer" containerID="d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"
Oct 30 18:55:36.474583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.474536    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:36.474921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.474886    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:36.475654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.475631    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "kube-api-access-8bbll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:36.476753 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.476729    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy" pod "2fc997a6-a405-4237-8a5d-4170e806d0fa" (UID: "2fc997a6-a405-4237-8a5d-4170e806d0fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:36.507776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.507751    5065 scope.go:122] "RemoveContainer" containerID="4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"
Oct 30 18:55:36.508147 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:36.508125    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609\": container with ID starting with 4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609 not found: ID does not exist" containerID="4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"
Oct 30 18:55:36.508232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.508155    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609"} err="failed to get container status \"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609\": rpc error: code = NotFound desc = could not find container \"4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609\": container with ID starting with 4721ab5b8c5eb09b242beb4eb3e78e0efdb666ce95c78464e74e155258722609 not found: ID does not exist"
Oct 30 18:55:36.508232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.508173    5065 scope.go:122] "RemoveContainer" containerID="d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"
Oct 30 18:55:36.508455 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:36.508440    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909\": container with ID starting with d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909 not found: ID does not exist" containerID="d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"
Oct 30 18:55:36.508501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.508459    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909"} err="failed to get container status \"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909\": rpc error: code = NotFound desc = could not find container \"d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909\": container with ID starting with d856cc1289ed73428ebb3fce093cc43b656653692c7582eaa852d8d07445f909 not found: ID does not exist"
Oct 30 18:55:36.572499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572473    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572499 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572497    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572507    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572515    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572524    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc997a6-a405-4237-8a5d-4170e806d0fa-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572531    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572541    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbll\" (UniqueName: \"kubernetes.io/projected/2fc997a6-a405-4237-8a5d-4170e806d0fa-kube-api-access-8bbll\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572549    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.572644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.572557    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc997a6-a405-4237-8a5d-4170e806d0fa-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.800410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.800375    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:55:36.806676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:36.806652    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f2ceb5-openshift-default-6ff58dcfd5-wnsfx"]
Oct 30 18:55:37.014573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.013161    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" podUID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" containerName="payload-pre-processing" containerID="cri-o://424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794" gracePeriod=25
Oct 30 18:55:37.017602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.017146    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" podUID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" containerName="payload-processing" containerID="cri-o://b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a" gracePeriod=25
Oct 30 18:55:37.373493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.373467    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:55:37.377211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.377190    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:55:37.396368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.396327    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\" (UniqueName: \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\") pod \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") "
Oct 30 18:55:37.396617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.396594    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\") pod \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\" (UID: \"fc104eac-5018-4ba7-b042-b7fd075f1fdb\") "
Oct 30 18:55:37.397054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.397029    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume" pod "fc104eac-5018-4ba7-b042-b7fd075f1fdb" (UID: "fc104eac-5018-4ba7-b042-b7fd075f1fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:37.397292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.397270    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc104eac-5018-4ba7-b042-b7fd075f1fdb-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:37.411134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.404171    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4" pod "fc104eac-5018-4ba7-b042-b7fd075f1fdb" (UID: "fc104eac-5018-4ba7-b042-b7fd075f1fdb"). InnerVolumeSpecName "kube-api-access-8sbm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:37.469808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.469782    5065 generic.go:356] "Generic (PLEG): container finished" podID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" containerID="b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a" exitCode=0
Oct 30 18:55:37.469962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.469833    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"
Oct 30 18:55:37.469962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.469860    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" event={"ID":"fc104eac-5018-4ba7-b042-b7fd075f1fdb","Type":"ContainerDied","Data":"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"}
Oct 30 18:55:37.469962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.469910    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp" event={"ID":"fc104eac-5018-4ba7-b042-b7fd075f1fdb","Type":"ContainerDied","Data":"296eec42519e9f3ff33f280b0af5f5872dad474ce0da2fc65e7d2bbcae6c92af"}
Oct 30 18:55:37.469962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.469940    5065 scope.go:122] "RemoveContainer" containerID="b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"
Oct 30 18:55:37.472914 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.472887    5065 generic.go:356] "Generic (PLEG): container finished" podID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" containerID="424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794" exitCode=0
Oct 30 18:55:37.473018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.472938    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"
Oct 30 18:55:37.473077 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.473017    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" event={"ID":"13e2f9f2-810f-4dac-b56d-35d0618b17fe","Type":"ContainerDied","Data":"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"}
Oct 30 18:55:37.473077 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.473072    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc" event={"ID":"13e2f9f2-810f-4dac-b56d-35d0618b17fe","Type":"ContainerDied","Data":"7b4c859775057410bc675306cc390d966d4c6e3bdeb1a0c2dd7435d0d8c53ecb"}
Oct 30 18:55:37.497932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.497904    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\" (UniqueName: \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\") pod \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") "
Oct 30 18:55:37.498206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.498176    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\") pod \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\" (UID: \"13e2f9f2-810f-4dac-b56d-35d0618b17fe\") "
Oct 30 18:55:37.498630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.498599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume" pod "13e2f9f2-810f-4dac-b56d-35d0618b17fe" (UID: "13e2f9f2-810f-4dac-b56d-35d0618b17fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:37.499406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.499380    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbm4\" (UniqueName: \"kubernetes.io/projected/fc104eac-5018-4ba7-b042-b7fd075f1fdb-kube-api-access-8sbm4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:37.499406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.499411    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e2f9f2-810f-4dac-b56d-35d0618b17fe-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:37.501131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.501065    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql" pod "13e2f9f2-810f-4dac-b56d-35d0618b17fe" (UID: "13e2f9f2-810f-4dac-b56d-35d0618b17fe"). InnerVolumeSpecName "kube-api-access-z9dql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:37.503018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.502992    5065 scope.go:122] "RemoveContainer" containerID="b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"
Oct 30 18:55:37.503403 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:37.503381    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a\": container with ID starting with b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a not found: ID does not exist" containerID="b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"
Oct 30 18:55:37.503469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.503416    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a"} err="failed to get container status \"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a\": rpc error: code = NotFound desc = could not find container \"b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a\": container with ID starting with b2ef88d0014dd77f01b20be9f9fe6b3cc6b6c28d92530c1e10e80b177d73b30a not found: ID does not exist"
Oct 30 18:55:37.503469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.503438    5065 scope.go:122] "RemoveContainer" containerID="424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"
Oct 30 18:55:37.516735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.516710    5065 scope.go:122] "RemoveContainer" containerID="424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"
Oct 30 18:55:37.517056 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:37.517030    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794\": container with ID starting with 424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794 not found: ID does not exist" containerID="424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"
Oct 30 18:55:37.517153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.517067    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794"} err="failed to get container status \"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794\": rpc error: code = NotFound desc = could not find container \"424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794\": container with ID starting with 424811a852027024c6fd307029785f070b3826022217fef9163c7563962c0794 not found: ID does not exist"
Oct 30 18:55:37.527310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.527291    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:55:37.533735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.533714    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9d42ff-6988845d64-m92sp"]
Oct 30 18:55:37.601375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.601309    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9dql\" (UniqueName: \"kubernetes.io/projected/13e2f9f2-810f-4dac-b56d-35d0618b17fe-kube-api-access-z9dql\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:37.805290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.805260    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:55:37.809229 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.809204    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9d42ff-85d8b84dc5-rnzcc"]
Oct 30 18:55:37.912173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.911460    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" path="/var/lib/kubelet/pods/13e2f9f2-810f-4dac-b56d-35d0618b17fe/volumes"
Oct 30 18:55:37.916011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.915971    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" path="/var/lib/kubelet/pods/2fc997a6-a405-4237-8a5d-4170e806d0fa/volumes"
Oct 30 18:55:37.916984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:37.916955    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" path="/var/lib/kubelet/pods/fc104eac-5018-4ba7-b042-b7fd075f1fdb/volumes"
Oct 30 18:55:44.785256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:44.785210    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"]
Oct 30 18:55:44.799005 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:44.798968    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-59f140-rbsv4"]
Oct 30 18:55:45.319253 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.319217    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"]
Oct 30 18:55:45.320630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.320604    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" containerName="payload-processing"
Oct 30 18:55:45.320802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.320789    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" containerName="payload-processing"
Oct 30 18:55:45.320902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.320892    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a52636fc-06de-4196-9599-75f8ff82fbbc" containerName="revoke-keys"
Oct 30 18:55:45.320973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.320963    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52636fc-06de-4196-9599-75f8ff82fbbc" containerName="revoke-keys"
Oct 30 18:55:45.321054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.321045    5065 cpu_manager.go:414] "containerMap: removing container" podUID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" containerName="payload-pre-processing"
Oct 30 18:55:45.321213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.321199    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" containerName="payload-pre-processing"
Oct 30 18:55:45.321310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.321300    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.321461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.321449    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.336201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.336168    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.336402 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.336386    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.336797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.336779    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52636fc-06de-4196-9599-75f8ff82fbbc" containerName="revoke-keys"
Oct 30 18:55:45.336912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.336900    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e2f9f2-810f-4dac-b56d-35d0618b17fe" containerName="payload-pre-processing"
Oct 30 18:55:45.337004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.336994    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.337112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.337100    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc997a6-a405-4237-8a5d-4170e806d0fa" containerName="istio-proxy"
Oct 30 18:55:45.337237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.337226    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc104eac-5018-4ba7-b042-b7fd075f1fdb" containerName="payload-processing"
Oct 30 18:55:45.343389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.343367    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.358119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.356931    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:45.358423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.358392    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:55:45.361870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.361851    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"]
Oct 30 18:55:45.425654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.423170    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.425654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.423251    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.425654 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.423495    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lq2l\" (UniqueName: \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.524903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.524863    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lq2l\" (UniqueName: \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.525699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.525626    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.526483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.526461    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.527685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.527507    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.529282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.529258    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.539267 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.539244    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lq2l\" (UniqueName: \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\") pod \"maas-api-revoke-sub-4f973f20-jlmjp\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.716801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.716721    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:45.902250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:45.901614    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52636fc-06de-4196-9599-75f8ff82fbbc" path="/var/lib/kubelet/pods/a52636fc-06de-4196-9599-75f8ff82fbbc/volumes"
Oct 30 18:55:46.054544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:46.054512    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"]
Oct 30 18:55:46.056578 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:46.056550    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded24ee1a_c381_4735_b749_34a5dcdf5619.slice/crio-a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98 WatchSource:0}: Error finding container a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98: Status 404 returned error can't find the container with id a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98
Oct 30 18:55:46.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:46.594024    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp" event={"ID":"ed24ee1a-c381-4735-b749-34a5dcdf5619","Type":"ContainerStarted","Data":"6986ec3bdda1dc136920fe69fa8db8a581803f2b376209fe458c1369de624eaf"}
Oct 30 18:55:46.594312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:46.594150    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp" event={"ID":"ed24ee1a-c381-4735-b749-34a5dcdf5619","Type":"ContainerStarted","Data":"a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98"}
Oct 30 18:55:46.620662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:46.620612    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp" podStartSLOduration=1.6205996040000001 podStartE2EDuration="1.620599604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:46.616121143 +0000 UTC m=+1366.879586092" watchObservedRunningTime="2026-10-30 18:55:46.620599604 +0000 UTC m=+1366.884064551"
Oct 30 18:55:47.610408 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:47.610359    5065 generic.go:356] "Generic (PLEG): container finished" podID="ed24ee1a-c381-4735-b749-34a5dcdf5619" containerID="6986ec3bdda1dc136920fe69fa8db8a581803f2b376209fe458c1369de624eaf" exitCode=0
Oct 30 18:55:47.611331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:47.610483    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp" event={"ID":"ed24ee1a-c381-4735-b749-34a5dcdf5619","Type":"ContainerDied","Data":"6986ec3bdda1dc136920fe69fa8db8a581803f2b376209fe458c1369de624eaf"}
Oct 30 18:55:48.838485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.838461    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:48.876363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.876285    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\" (UniqueName: \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\") pod \"ed24ee1a-c381-4735-b749-34a5dcdf5619\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") "
Oct 30 18:55:48.876521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.876446    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\") pod \"ed24ee1a-c381-4735-b749-34a5dcdf5619\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") "
Oct 30 18:55:48.876593 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.876525    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\") pod \"ed24ee1a-c381-4735-b749-34a5dcdf5619\" (UID: \"ed24ee1a-c381-4735-b749-34a5dcdf5619\") "
Oct 30 18:55:48.877180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.877150    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca" pod "ed24ee1a-c381-4735-b749-34a5dcdf5619" (UID: "ed24ee1a-c381-4735-b749-34a5dcdf5619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:48.883529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.883499    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l" pod "ed24ee1a-c381-4735-b749-34a5dcdf5619" (UID: "ed24ee1a-c381-4735-b749-34a5dcdf5619"). InnerVolumeSpecName "kube-api-access-7lq2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:48.885626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.885601    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca" pod "ed24ee1a-c381-4735-b749-34a5dcdf5619" (UID: "ed24ee1a-c381-4735-b749-34a5dcdf5619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:48.978590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.978566    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:48.978590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.978591    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed24ee1a-c381-4735-b749-34a5dcdf5619-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:48.978743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:48.978602    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lq2l\" (UniqueName: \"kubernetes.io/projected/ed24ee1a-c381-4735-b749-34a5dcdf5619-kube-api-access-7lq2l\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.631501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:49.631473    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"
Oct 30 18:55:49.631501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:49.631486    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp" event={"ID":"ed24ee1a-c381-4735-b749-34a5dcdf5619","Type":"ContainerDied","Data":"a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98"}
Oct 30 18:55:49.631734 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:49.631533    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a28caf66cd05d76f9bc8f26b90575eed30c25bb9da803d8443cb9bb8d9b71d98"
Oct 30 18:55:51.099358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:51.099321    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9_12ee5981-c096-4bf6-9418-7a5d5c0e1100/payload-processing/0.log"
Oct 30 18:55:51.303075 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:51.303047    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 18:55:51.517401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:51.517376    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 18:55:52.511248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.511218    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"]
Oct 30 18:55:52.511951 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.511933    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ed24ee1a-c381-4735-b749-34a5dcdf5619" containerName="revoke-keys"
Oct 30 18:55:52.512002 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.511954    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed24ee1a-c381-4735-b749-34a5dcdf5619" containerName="revoke-keys"
Oct 30 18:55:52.512115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.512099    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed24ee1a-c381-4735-b749-34a5dcdf5619" containerName="revoke-keys"
Oct 30 18:55:52.516053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.516023    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.522472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.522447    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"]
Oct 30 18:55:52.523675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.523654    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:52.525066 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.525048    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:55:52.637348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.637320    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.637511 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.637491    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rz8r\" (UniqueName: \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.637635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.637619    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.746009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.744516    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.746009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.744763    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rz8r\" (UniqueName: \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.746009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.744947    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.755185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.755134    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.762660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.762582    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.773056 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.773021    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rz8r\" (UniqueName: \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\") pod \"maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:52.836637 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:52.836600    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:53.055813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:53.055732    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:55:53.056009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:53.055986    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" containerID="cri-o://ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f" gracePeriod=30
Oct 30 18:55:53.074164 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:53.074134    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:55:53.250116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:53.246233    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"]
Oct 30 18:55:53.667069 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:53.667035    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp" event={"ID":"642412e6-2173-408a-b9c9-1a508d72e305","Type":"ContainerStarted","Data":"03b2b1a81899f98c12fb7a6df4ea2b167dbe39bebfac8134a64f8969459a43da"}
Oct 30 18:55:54.681049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:54.680955    5065 generic.go:356] "Generic (PLEG): container finished" podID="642412e6-2173-408a-b9c9-1a508d72e305" containerID="8b65840d9a06a899d51c84dd9040ae4f7bbc79b4e6d5921830e9fc911cbd6872" exitCode=0
Oct 30 18:55:54.681049 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:54.680997    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp" event={"ID":"642412e6-2173-408a-b9c9-1a508d72e305","Type":"ContainerDied","Data":"8b65840d9a06a899d51c84dd9040ae4f7bbc79b4e6d5921830e9fc911cbd6872"}
Oct 30 18:55:55.386855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.386809    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 18:55:55.387067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.386878    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:55:55.735145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.735116    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"]
Oct 30 18:55:55.740917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.740897    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.787560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.787526    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"]
Oct 30 18:55:55.825826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.825796    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.825990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.825970    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5d7\" (UniqueName: \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.826135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.826073    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.875521 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.875497    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:55.927320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.927280    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5d7\" (UniqueName: \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.927500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.927347    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.927578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.927532    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.928498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.928348    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.931545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.931061    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:55.959383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:55.959329    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5d7\" (UniqueName: \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\") pod \"maas-api-revoke-sub-9872e69a-rt5m4\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:56.028260 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.028238    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\") pod \"642412e6-2173-408a-b9c9-1a508d72e305\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") "
Oct 30 18:55:56.028367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.028285    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\" (UniqueName: \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\") pod \"642412e6-2173-408a-b9c9-1a508d72e305\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") "
Oct 30 18:55:56.028367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.028358    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\") pod \"642412e6-2173-408a-b9c9-1a508d72e305\" (UID: \"642412e6-2173-408a-b9c9-1a508d72e305\") "
Oct 30 18:55:56.028552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.028530    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca" pod "642412e6-2173-408a-b9c9-1a508d72e305" (UID: "642412e6-2173-408a-b9c9-1a508d72e305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:56.028955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.028929    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:56.030551 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.030532    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r" pod "642412e6-2173-408a-b9c9-1a508d72e305" (UID: "642412e6-2173-408a-b9c9-1a508d72e305"). InnerVolumeSpecName "kube-api-access-5rz8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:56.030732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.030718    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca" pod "642412e6-2173-408a-b9c9-1a508d72e305" (UID: "642412e6-2173-408a-b9c9-1a508d72e305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:56.055678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.055657    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:56.130926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.130894    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rz8r\" (UniqueName: \"kubernetes.io/projected/642412e6-2173-408a-b9c9-1a508d72e305-kube-api-access-5rz8r\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:56.133736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.130932    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642412e6-2173-408a-b9c9-1a508d72e305-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:56.352328 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:56.352301    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e93cf41_7919_4197_8fdc_ea8b8ed53310.slice/crio-acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447 WatchSource:0}: Error finding container acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447: Status 404 returned error can't find the container with id acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447
Oct 30 18:55:56.352567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.352545    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"]
Oct 30 18:55:56.702763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.702726    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4" event={"ID":"0e93cf41-7919-4197-8fdc-ea8b8ed53310","Type":"ContainerStarted","Data":"acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447"}
Oct 30 18:55:56.704755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.704721    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp" event={"ID":"642412e6-2173-408a-b9c9-1a508d72e305","Type":"ContainerDied","Data":"03b2b1a81899f98c12fb7a6df4ea2b167dbe39bebfac8134a64f8969459a43da"}
Oct 30 18:55:56.704920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.704771    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b2b1a81899f98c12fb7a6df4ea2b167dbe39bebfac8134a64f8969459a43da"
Oct 30 18:55:56.705036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:56.705009    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"
Oct 30 18:55:57.716070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.715984    5065 generic.go:356] "Generic (PLEG): container finished" podID="0e93cf41-7919-4197-8fdc-ea8b8ed53310" containerID="6bc635941028ef8380da24642f81bd8f3f72413e1b99d3c61adb262e9781ee44" exitCode=0
Oct 30 18:55:57.716549 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.716069    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4" event={"ID":"0e93cf41-7919-4197-8fdc-ea8b8ed53310","Type":"ContainerDied","Data":"6bc635941028ef8380da24642f81bd8f3f72413e1b99d3c61adb262e9781ee44"}
Oct 30 18:55:57.970556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.970486    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"]
Oct 30 18:55:57.971223 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.971206    5065 cpu_manager.go:414] "containerMap: removing container" podUID="642412e6-2173-408a-b9c9-1a508d72e305" containerName="revoke-keys"
Oct 30 18:55:57.971286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.971224    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642412e6-2173-408a-b9c9-1a508d72e305" containerName="revoke-keys"
Oct 30 18:55:57.971349 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.971339    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="642412e6-2173-408a-b9c9-1a508d72e305" containerName="revoke-keys"
Oct 30 18:55:57.974909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.974893    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:57.990915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:57.987663    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"]
Oct 30 18:55:58.061604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.061560    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.061759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.061658    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.061825 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.061755    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j85c\" (UniqueName: \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.162661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.162630    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.162799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.162754    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.162885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.162845    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j85c\" (UniqueName: \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.163478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.163450    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.166305 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.166271    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.191500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.191449    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j85c\" (UniqueName: \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\") pod \"maas-api-revoke-sub-7307a27a-sd7hz\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.326178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.321487    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:55:58.432536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.432468    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:58.576863 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.576830    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.577513 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.577491    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.577991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.577975    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.578129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.577406    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:58.578211 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.577893    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:58.578358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.578341    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.578509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.578497    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.578612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.578600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.578802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.578788    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.578909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.578895    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.579054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.579040    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\") pod \"0788b89e-2689-42bf-b985-d4aeddd8dc62\" (UID: \"0788b89e-2689-42bf-b985-d4aeddd8dc62\") "
Oct 30 18:55:58.579822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.579799    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0788b89e-2689-42bf-b985-d4aeddd8dc62-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.579936 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.579925    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.583133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.582915    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "kube-api-access-t2npn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:58.583133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.583040    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:58.583486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.583309    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:58.583741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.583592    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:58.583898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.583856    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:58.586233 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.586196    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:58.586884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.586859    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token" pod "0788b89e-2689-42bf-b985-d4aeddd8dc62" (UID: "0788b89e-2689-42bf-b985-d4aeddd8dc62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:58.659960 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:55:58.659935    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2701c4_0c1d_4640_87a6_87da13ba67d5.slice/crio-b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c WatchSource:0}: Error finding container b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c: Status 404 returned error can't find the container with id b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c
Oct 30 18:55:58.660345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.660323    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"]
Oct 30 18:55:58.681644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681612    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681644 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681644    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681654    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2npn\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-kube-api-access-t2npn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681664    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681673    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681682    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.681782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.681690    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0788b89e-2689-42bf-b985-d4aeddd8dc62-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.727687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.727653    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz" event={"ID":"5d2701c4-0c1d-4640-87a6-87da13ba67d5","Type":"ContainerStarted","Data":"b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c"}
Oct 30 18:55:58.729800 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.729779    5065 generic.go:356] "Generic (PLEG): container finished" podID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerID="ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f" exitCode=0
Oct 30 18:55:58.729921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.729848    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"
Oct 30 18:55:58.729921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.729869    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerDied","Data":"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"}
Oct 30 18:55:58.730045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.729918    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp" event={"ID":"0788b89e-2689-42bf-b985-d4aeddd8dc62","Type":"ContainerDied","Data":"b5ba0eab09260aebf29b674aa704ca04c09368128d1f23106ab5d41bca439203"}
Oct 30 18:55:58.730045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.729949    5065 scope.go:122] "RemoveContainer" containerID="ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"
Oct 30 18:55:58.745859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.745841    5065 scope.go:122] "RemoveContainer" containerID="794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"
Oct 30 18:55:58.760796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.760774    5065 scope.go:122] "RemoveContainer" containerID="ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"
Oct 30 18:55:58.761131 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:58.761102    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f\": container with ID starting with ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f not found: ID does not exist" containerID="ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"
Oct 30 18:55:58.761296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.761139    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f"} err="failed to get container status \"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f\": rpc error: code = NotFound desc = could not find container \"ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f\": container with ID starting with ce2f162120a7031bb8983450f596723fe0fe0bc850595bef2a6315ba0cd4e06f not found: ID does not exist"
Oct 30 18:55:58.761296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.761162    5065 scope.go:122] "RemoveContainer" containerID="794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"
Oct 30 18:55:58.761538 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:55:58.761517    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359\": container with ID starting with 794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359 not found: ID does not exist" containerID="794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"
Oct 30 18:55:58.761607 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.761546    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359"} err="failed to get container status \"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359\": rpc error: code = NotFound desc = could not find container \"794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359\": container with ID starting with 794e0304fab0a8a8706bade95c293a40bdfd9b9e2ad16edc82b50662a9e60359 not found: ID does not exist"
Oct 30 18:55:58.767106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.766716    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:55:58.773101 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.772463    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-59f140-openshift-default-5f6bdb4bfc-t7rwp"]
Oct 30 18:55:58.904577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.904550    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:58.985378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.985348    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\" (UniqueName: \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\") pod \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") "
Oct 30 18:55:58.985548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.985404    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\") pod \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") "
Oct 30 18:55:58.985548 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.985454    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\") pod \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\" (UID: \"0e93cf41-7919-4197-8fdc-ea8b8ed53310\") "
Oct 30 18:55:58.985856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.985822    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca" pod "0e93cf41-7919-4197-8fdc-ea8b8ed53310" (UID: "0e93cf41-7919-4197-8fdc-ea8b8ed53310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:58.986108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.986070    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:58.988118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.988074    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7" pod "0e93cf41-7919-4197-8fdc-ea8b8ed53310" (UID: "0e93cf41-7919-4197-8fdc-ea8b8ed53310"). InnerVolumeSpecName "kube-api-access-7q5d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:58.988493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:58.988472    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca" pod "0e93cf41-7919-4197-8fdc-ea8b8ed53310" (UID: "0e93cf41-7919-4197-8fdc-ea8b8ed53310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:59.087906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.087878    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q5d7\" (UniqueName: \"kubernetes.io/projected/0e93cf41-7919-4197-8fdc-ea8b8ed53310-kube-api-access-7q5d7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:59.088012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.087910    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93cf41-7919-4197-8fdc-ea8b8ed53310-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:59.741932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.741898    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz" event={"ID":"5d2701c4-0c1d-4640-87a6-87da13ba67d5","Type":"ContainerStarted","Data":"3bef2f301786d79e27392639923977292aa6aa15669a3897d9360a03877962f2"}
Oct 30 18:55:59.743347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.743327    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"
Oct 30 18:55:59.743480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.743363    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4" event={"ID":"0e93cf41-7919-4197-8fdc-ea8b8ed53310","Type":"ContainerDied","Data":"acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447"}
Oct 30 18:55:59.743480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.743407    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd106019b010ed2d897090859f6f37df7f257c961debb250316772ea80f8447"
Oct 30 18:55:59.763307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.763270    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz" podStartSLOduration=2.763258117 podStartE2EDuration="2.763258117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:59.761478286 +0000 UTC m=+1380.024943243" watchObservedRunningTime="2026-10-30 18:55:59.763258117 +0000 UTC m=+1380.026723064"
Oct 30 18:55:59.892371 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:55:59.892346    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" path="/var/lib/kubelet/pods/0788b89e-2689-42bf-b985-d4aeddd8dc62/volumes"
Oct 30 18:56:00.026906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:00.026873    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"]
Oct 30 18:56:00.031308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:00.031280    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59bd1c3b-xt2ft"]
Oct 30 18:56:00.756100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:00.756055    5065 generic.go:356] "Generic (PLEG): container finished" podID="5d2701c4-0c1d-4640-87a6-87da13ba67d5" containerID="3bef2f301786d79e27392639923977292aa6aa15669a3897d9360a03877962f2" exitCode=0
Oct 30 18:56:00.756505 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:00.756141    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz" event={"ID":"5d2701c4-0c1d-4640-87a6-87da13ba67d5","Type":"ContainerDied","Data":"3bef2f301786d79e27392639923977292aa6aa15669a3897d9360a03877962f2"}
Oct 30 18:56:01.965726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:01.965697    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dd5c56-1026-4303-8825-6912c5dbce26" path="/var/lib/kubelet/pods/d5dd5c56-1026-4303-8825-6912c5dbce26/volumes"
Oct 30 18:56:02.010602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.010578    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:56:02.051617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.051539    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\") pod \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") "
Oct 30 18:56:02.051617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.051593    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\" (UniqueName: \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\") pod \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") "
Oct 30 18:56:02.051851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.051699    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\") pod \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\" (UID: \"5d2701c4-0c1d-4640-87a6-87da13ba67d5\") "
Oct 30 18:56:02.052055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.052029    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca" pod "5d2701c4-0c1d-4640-87a6-87da13ba67d5" (UID: "5d2701c4-0c1d-4640-87a6-87da13ba67d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:02.052300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.052283    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:02.054134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.054068    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca" pod "5d2701c4-0c1d-4640-87a6-87da13ba67d5" (UID: "5d2701c4-0c1d-4640-87a6-87da13ba67d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:02.054719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.054691    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c" pod "5d2701c4-0c1d-4640-87a6-87da13ba67d5" (UID: "5d2701c4-0c1d-4640-87a6-87da13ba67d5"). InnerVolumeSpecName "kube-api-access-2j85c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:02.153673 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.153644    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2701c4-0c1d-4640-87a6-87da13ba67d5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:02.153673 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.153671    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j85c\" (UniqueName: \"kubernetes.io/projected/5d2701c4-0c1d-4640-87a6-87da13ba67d5-kube-api-access-2j85c\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:02.775805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.775763    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz" event={"ID":"5d2701c4-0c1d-4640-87a6-87da13ba67d5","Type":"ContainerDied","Data":"b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c"}
Oct 30 18:56:02.775975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.775823    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b097033ba26cb0432198c142788d14eee7d3fa1e1f236938276924ea04fb1c3c"
Oct 30 18:56:02.775975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:02.775783    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"
Oct 30 18:56:05.845411 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.845380    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"]
Oct 30 18:56:05.846314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846291    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5d2701c4-0c1d-4640-87a6-87da13ba67d5" containerName="revoke-keys"
Oct 30 18:56:05.846314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846314    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2701c4-0c1d-4640-87a6-87da13ba67d5" containerName="revoke-keys"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846338    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846345    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846354    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846362    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846415    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0e93cf41-7919-4197-8fdc-ea8b8ed53310" containerName="revoke-keys"
Oct 30 18:56:05.846512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846422    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e93cf41-7919-4197-8fdc-ea8b8ed53310" containerName="revoke-keys"
Oct 30 18:56:05.846809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846563    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846578    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0788b89e-2689-42bf-b985-d4aeddd8dc62" containerName="istio-proxy"
Oct 30 18:56:05.846809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846598    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e93cf41-7919-4197-8fdc-ea8b8ed53310" containerName="revoke-keys"
Oct 30 18:56:05.846809 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.846613    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2701c4-0c1d-4640-87a6-87da13ba67d5" containerName="revoke-keys"
Oct 30 18:56:05.850709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.850690    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:05.863889 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.863860    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"]
Oct 30 18:56:05.910384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.910347    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:05.910551 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.910421    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wxg\" (UniqueName: \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:05.910640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.910618    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:05.945726 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.945697    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:05.945996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:05.945978    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:56:06.011795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.011765    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.011952 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.011842    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wxg\" (UniqueName: \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.012045 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.012024    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.012631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.012598    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.014630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.014600    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.033146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.033118    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wxg\" (UniqueName: \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\") pod \"maas-api-revoke-sub-47c2c241-wtghx\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.166791 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.166717    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:06.466768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.466726    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"]
Oct 30 18:56:06.466900 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:06.466770    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0d9fe9_a075_46a1_9f49_c974594d8d52.slice/crio-45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19 WatchSource:0}: Error finding container 45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19: Status 404 returned error can't find the container with id 45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19
Oct 30 18:56:06.835686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:06.835650    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx" event={"ID":"3e0d9fe9-a075-46a1-9f49-c974594d8d52","Type":"ContainerStarted","Data":"45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19"}
Oct 30 18:56:07.845567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:07.845526    5065 generic.go:356] "Generic (PLEG): container finished" podID="3e0d9fe9-a075-46a1-9f49-c974594d8d52" containerID="c29579dc1f9291179d8d8670ca4bd421f5e4c2bd19e61a5a8b093df00873eb9c" exitCode=0
Oct 30 18:56:07.845950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:07.845597    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx" event={"ID":"3e0d9fe9-a075-46a1-9f49-c974594d8d52","Type":"ContainerDied","Data":"c29579dc1f9291179d8d8670ca4bd421f5e4c2bd19e61a5a8b093df00873eb9c"}
Oct 30 18:56:08.482118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:08.475918    5065 scope.go:122] "RemoveContainer" containerID="f6d3f1c177c4092470f4384df5dd709fe648e4bd337ecb7f3656a80524274012"
Oct 30 18:56:08.528537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:08.528296    5065 scope.go:122] "RemoveContainer" containerID="90fda4db04b9fa5473520955f14e286daa46ccc1a517ce62b821d3805701364a"
Oct 30 18:56:08.587257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:08.587226    5065 scope.go:122] "RemoveContainer" containerID="d2723feb20bf4fd0caf6a11dd70a48bf1893ec42f25255b1a07466f7adcea5fe"
Oct 30 18:56:08.672344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:08.669431    5065 scope.go:122] "RemoveContainer" containerID="9418d8cee1ca4bb657ffe372ae2625ca909c4c9621874592ab031750bbf438f6"
Oct 30 18:56:10.155169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.150571    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:10.219113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.218576    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:10.219289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.219119    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx" event={"ID":"3e0d9fe9-a075-46a1-9f49-c974594d8d52","Type":"ContainerDied","Data":"45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19"}
Oct 30 18:56:10.219289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.219172    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45289b0557bce6a3ed963ef52aa2b5b4e5093e681c9669d6f0eb305351b9af19"
Oct 30 18:56:10.362167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.354547    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\") pod \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") "
Oct 30 18:56:10.362167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.354762    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\") pod \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") "
Oct 30 18:56:10.362167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.355059    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\" (UniqueName: \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\") pod \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\" (UID: \"3e0d9fe9-a075-46a1-9f49-c974594d8d52\") "
Oct 30 18:56:10.362167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.358856    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:10.362167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.360592    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Oct 30 18:56:10.371509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.371299    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca" pod "3e0d9fe9-a075-46a1-9f49-c974594d8d52" (UID: "3e0d9fe9-a075-46a1-9f49-c974594d8d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:10.530120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.527620    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.566123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.565570    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg" pod "3e0d9fe9-a075-46a1-9f49-c974594d8d52" (UID: "3e0d9fe9-a075-46a1-9f49-c974594d8d52"). InnerVolumeSpecName "kube-api-access-m5wxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:10.588109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.585244    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca" pod "3e0d9fe9-a075-46a1-9f49-c974594d8d52" (UID: "3e0d9fe9-a075-46a1-9f49-c974594d8d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:10.642150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.641387    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wxg\" (UniqueName: \"kubernetes.io/projected/3e0d9fe9-a075-46a1-9f49-c974594d8d52-kube-api-access-m5wxg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.642150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.641425    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0d9fe9-a075-46a1-9f49-c974594d8d52-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:10.789181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.784163    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:56:10.789181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.785403    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:10.789181 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.787473    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:56:10.819122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.816752    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:56:11.012561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:10.965698    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:11.013954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.003661    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:56:11.014325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.014267    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:11.239153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.239036    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:11.245491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.245465    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:56:11.355134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.343738    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"
Oct 30 18:56:11.359491 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.359455    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:11.364120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.362726    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:56:11.508469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.506976    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:56:11.536061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.511140    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:56:11.794024 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.793944    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"]
Oct 30 18:56:11.794705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.794684    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3e0d9fe9-a075-46a1-9f49-c974594d8d52" containerName="revoke-keys"
Oct 30 18:56:11.794705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.794701    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0d9fe9-a075-46a1-9f49-c974594d8d52" containerName="revoke-keys"
Oct 30 18:56:11.794923 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.794852    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0d9fe9-a075-46a1-9f49-c974594d8d52" containerName="revoke-keys"
Oct 30 18:56:11.798757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.798740    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.811574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.811546    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"]
Oct 30 18:56:11.864441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.864407    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2n4r\" (UniqueName: \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.864589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.864548    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.864692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.864673    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.893477 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.893455    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:11.893673 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.893534    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:56:11.966182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.966155    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.966342 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.966286    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2n4r\" (UniqueName: \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.966460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.966439    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.966932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.966907    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.969038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.969017    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:11.985390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:11.984186    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2n4r\" (UniqueName: \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\") pod \"maas-api-revoke-sub-7d684db0-xd5wj\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:12.124356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.124278    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:12.226853 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.226821    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"]
Oct 30 18:56:12.233845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.233822    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.251829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.251259    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"]
Oct 30 18:56:12.372840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.372813    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.373000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.372850    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.373000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.372877    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbr9f\" (UniqueName: \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.474939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.474835    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.474939 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.474904    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.475176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.474944    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbr9f\" (UniqueName: \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.475594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.475576    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.477357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.477329    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.501556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.501524    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"]
Oct 30 18:56:12.503774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.503750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbr9f\" (UniqueName: \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\") pod \"maas-api-revoke-sub-9a8c28ff-h79rx\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.505341 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:12.505313    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2e397e_6167_49c5_a230_ba39ba720504.slice/crio-32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e WatchSource:0}: Error finding container 32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e: Status 404 returned error can't find the container with id 32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e
Oct 30 18:56:12.558010 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.557986    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:12.863950 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:12.863924    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e4bc45_1c84_421b_b763_3d28f2a6bdac.slice/crio-c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe WatchSource:0}: Error finding container c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe: Status 404 returned error can't find the container with id c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe
Oct 30 18:56:12.864424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:12.864397    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"]
Oct 30 18:56:13.363493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.363457    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx" event={"ID":"86e4bc45-1c84-421b-b763-3d28f2a6bdac","Type":"ContainerStarted","Data":"b6688ddb0aac3cab6120a0bb694769ebf306755a9a83ad2f6f4727a8cae83fc5"}
Oct 30 18:56:13.363867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.363554    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx" event={"ID":"86e4bc45-1c84-421b-b763-3d28f2a6bdac","Type":"ContainerStarted","Data":"c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe"}
Oct 30 18:56:13.365240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.365219    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj" event={"ID":"9f2e397e-6167-49c5-a230-ba39ba720504","Type":"ContainerStarted","Data":"cb61070e4af50a11a41e8887b9eaf20f8ac0430d1fde484a2be8bf199efab904"}
Oct 30 18:56:13.365323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.365256    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj" event={"ID":"9f2e397e-6167-49c5-a230-ba39ba720504","Type":"ContainerStarted","Data":"32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e"}
Oct 30 18:56:13.388348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.388271    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx" podStartSLOduration=1.388257039 podStartE2EDuration="1.388257039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:13.383511709 +0000 UTC m=+1393.646976689" watchObservedRunningTime="2026-10-30 18:56:13.388257039 +0000 UTC m=+1393.651722039"
Oct 30 18:56:13.414378 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:13.414332    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj" podStartSLOduration=2.414318133 podStartE2EDuration="2.414318133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:13.413422356 +0000 UTC m=+1393.676887306" watchObservedRunningTime="2026-10-30 18:56:13.414318133 +0000 UTC m=+1393.677783081"
Oct 30 18:56:14.375388 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:14.375356    5065 generic.go:356] "Generic (PLEG): container finished" podID="9f2e397e-6167-49c5-a230-ba39ba720504" containerID="cb61070e4af50a11a41e8887b9eaf20f8ac0430d1fde484a2be8bf199efab904" exitCode=0
Oct 30 18:56:14.375829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:14.375448    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj" event={"ID":"9f2e397e-6167-49c5-a230-ba39ba720504","Type":"ContainerDied","Data":"cb61070e4af50a11a41e8887b9eaf20f8ac0430d1fde484a2be8bf199efab904"}
Oct 30 18:56:14.377270 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:14.377248    5065 generic.go:356] "Generic (PLEG): container finished" podID="86e4bc45-1c84-421b-b763-3d28f2a6bdac" containerID="b6688ddb0aac3cab6120a0bb694769ebf306755a9a83ad2f6f4727a8cae83fc5" exitCode=0
Oct 30 18:56:14.377365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:14.377333    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx" event={"ID":"86e4bc45-1c84-421b-b763-3d28f2a6bdac","Type":"ContainerDied","Data":"b6688ddb0aac3cab6120a0bb694769ebf306755a9a83ad2f6f4727a8cae83fc5"}
Oct 30 18:56:15.648464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.648422    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:15.671310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.671284    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.860810    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\" (UniqueName: \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\") pod \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.860884    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\") pod \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.861043    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\") pod \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\" (UID: \"86e4bc45-1c84-421b-b763-3d28f2a6bdac\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.861122    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\") pod \"9f2e397e-6167-49c5-a230-ba39ba720504\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.861178    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\" (UniqueName: \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\") pod \"9f2e397e-6167-49c5-a230-ba39ba720504\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.861207    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\") pod \"9f2e397e-6167-49c5-a230-ba39ba720504\" (UID: \"9f2e397e-6167-49c5-a230-ba39ba720504\") "
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.862581    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca" pod "9f2e397e-6167-49c5-a230-ba39ba720504" (UID: "9f2e397e-6167-49c5-a230-ba39ba720504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:15.863988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.863733    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca" pod "86e4bc45-1c84-421b-b763-3d28f2a6bdac" (UID: "86e4bc45-1c84-421b-b763-3d28f2a6bdac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:15.867676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.867641    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca" pod "86e4bc45-1c84-421b-b763-3d28f2a6bdac" (UID: "86e4bc45-1c84-421b-b763-3d28f2a6bdac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:15.868000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.867718    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f" pod "86e4bc45-1c84-421b-b763-3d28f2a6bdac" (UID: "86e4bc45-1c84-421b-b763-3d28f2a6bdac"). InnerVolumeSpecName "kube-api-access-tbr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:15.869901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.869870    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca" pod "9f2e397e-6167-49c5-a230-ba39ba720504" (UID: "9f2e397e-6167-49c5-a230-ba39ba720504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:15.870459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.870434    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r" pod "9f2e397e-6167-49c5-a230-ba39ba720504" (UID: "9f2e397e-6167-49c5-a230-ba39ba720504"). InnerVolumeSpecName "kube-api-access-j2n4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:15.962888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.962858    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2n4r\" (UniqueName: \"kubernetes.io/projected/9f2e397e-6167-49c5-a230-ba39ba720504-kube-api-access-j2n4r\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:15.963017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.962977    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:15.963017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.962996    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbr9f\" (UniqueName: \"kubernetes.io/projected/86e4bc45-1c84-421b-b763-3d28f2a6bdac-kube-api-access-tbr9f\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:15.963017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.963012    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:15.963175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.963025    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e4bc45-1c84-421b-b763-3d28f2a6bdac-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:15.963175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:15.963034    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e397e-6167-49c5-a230-ba39ba720504-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:16.396507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.396480    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"
Oct 30 18:56:16.396687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.396519    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx" event={"ID":"86e4bc45-1c84-421b-b763-3d28f2a6bdac","Type":"ContainerDied","Data":"c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe"}
Oct 30 18:56:16.396687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.396600    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c573ee054f327309c898f8b1b5d213cdc79d0e41ca8b2121f47b5172dc8d1ebe"
Oct 30 18:56:16.398330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.398265    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"
Oct 30 18:56:16.398330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.398285    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj" event={"ID":"9f2e397e-6167-49c5-a230-ba39ba720504","Type":"ContainerDied","Data":"32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e"}
Oct 30 18:56:16.398330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:16.398332    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32876af6b90b8936a96fcf5d4e81c8f14e6cb821be43ca11f5ceba9692b2187e"
Oct 30 18:56:20.655788 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:20.655755    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:56:20.656461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:20.656003    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" podUID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" containerName="payload-processing" containerID="cri-o://b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af" gracePeriod=30
Oct 30 18:56:20.662624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:20.662596    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:56:20.662869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:20.662812    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" podUID="ab0a2ebd-912c-4c20-89af-cd10830e7118" containerName="payload-pre-processing" containerID="cri-o://54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610" gracePeriod=30
Oct 30 18:56:21.201171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.201107    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"]
Oct 30 18:56:21.208311 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.208271    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9ceefc-v6clp"]
Oct 30 18:56:21.270188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.270114    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:56:21.315050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.315025    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:56:21.418134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418052    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\") pod \"ab0a2ebd-912c-4c20-89af-cd10830e7118\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") "
Oct 30 18:56:21.418134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418124    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\" (UniqueName: \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\") pod \"ab0a2ebd-912c-4c20-89af-cd10830e7118\" (UID: \"ab0a2ebd-912c-4c20-89af-cd10830e7118\") "
Oct 30 18:56:21.418361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418242    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\" (UniqueName: \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\") pod \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") "
Oct 30 18:56:21.418361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418311    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\") pod \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\" (UID: \"12ee5981-c096-4bf6-9418-7a5d5c0e1100\") "
Oct 30 18:56:21.418540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418456    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume" pod "ab0a2ebd-912c-4c20-89af-cd10830e7118" (UID: "ab0a2ebd-912c-4c20-89af-cd10830e7118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:21.418801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.418771    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume" pod "12ee5981-c096-4bf6-9418-7a5d5c0e1100" (UID: "12ee5981-c096-4bf6-9418-7a5d5c0e1100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:21.419202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.419180    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0a2ebd-912c-4c20-89af-cd10830e7118-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.419307 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.419208    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ee5981-c096-4bf6-9418-7a5d5c0e1100-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.420628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.420606    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf" pod "ab0a2ebd-912c-4c20-89af-cd10830e7118" (UID: "ab0a2ebd-912c-4c20-89af-cd10830e7118"). InnerVolumeSpecName "kube-api-access-xkpgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:21.420717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.420678    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r" pod "12ee5981-c096-4bf6-9418-7a5d5c0e1100" (UID: "12ee5981-c096-4bf6-9418-7a5d5c0e1100"). InnerVolumeSpecName "kube-api-access-m6v2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:21.444506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.444479    5065 generic.go:356] "Generic (PLEG): container finished" podID="ab0a2ebd-912c-4c20-89af-cd10830e7118" containerID="54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610" exitCode=0
Oct 30 18:56:21.444615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.444529    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"
Oct 30 18:56:21.444615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.444565    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" event={"ID":"ab0a2ebd-912c-4c20-89af-cd10830e7118","Type":"ContainerDied","Data":"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"}
Oct 30 18:56:21.444714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.444627    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7" event={"ID":"ab0a2ebd-912c-4c20-89af-cd10830e7118","Type":"ContainerDied","Data":"b6253466c4b9ae788209f6f990f92ae32b289e57068a7279f6c89817446b3043"}
Oct 30 18:56:21.444714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.444657    5065 scope.go:122] "RemoveContainer" containerID="54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"
Oct 30 18:56:21.446756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.446722    5065 generic.go:356] "Generic (PLEG): container finished" podID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" containerID="b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af" exitCode=0
Oct 30 18:56:21.446861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.446798    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"
Oct 30 18:56:21.447030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.446999    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" event={"ID":"12ee5981-c096-4bf6-9418-7a5d5c0e1100","Type":"ContainerDied","Data":"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"}
Oct 30 18:56:21.447150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.447055    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9" event={"ID":"12ee5981-c096-4bf6-9418-7a5d5c0e1100","Type":"ContainerDied","Data":"92bef74483b283b147886b9c501566f07d7684296ad749099136a1353e343aa2"}
Oct 30 18:56:21.457188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.457168    5065 scope.go:122] "RemoveContainer" containerID="54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"
Oct 30 18:56:21.457456 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:56:21.457435    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610\": container with ID starting with 54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610 not found: ID does not exist" containerID="54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"
Oct 30 18:56:21.457541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.457467    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610"} err="failed to get container status \"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610\": rpc error: code = NotFound desc = could not find container \"54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610\": container with ID starting with 54b083f639fb70a36cf393e8b49e1715d0203c7a14a89397632bce817741e610 not found: ID does not exist"
Oct 30 18:56:21.457541 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.457489    5065 scope.go:122] "RemoveContainer" containerID="b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"
Oct 30 18:56:21.468689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.468666    5065 scope.go:122] "RemoveContainer" containerID="b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"
Oct 30 18:56:21.469048 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:56:21.469021    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af\": container with ID starting with b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af not found: ID does not exist" containerID="b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"
Oct 30 18:56:21.469165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.469058    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af"} err="failed to get container status \"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af\": rpc error: code = NotFound desc = could not find container \"b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af\": container with ID starting with b251fcbbb149490c45cedbd65a25db022941c3d39b8168143f936132ce2138af not found: ID does not exist"
Oct 30 18:56:21.479648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.479624    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:56:21.485006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.484981    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-59f8c7b747-nrxv7"]
Oct 30 18:56:21.501308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.501280    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:56:21.504776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.504757    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-6cb66ddb64-h9zc9"]
Oct 30 18:56:21.520413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.520341    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6v2r\" (UniqueName: \"kubernetes.io/projected/12ee5981-c096-4bf6-9418-7a5d5c0e1100-kube-api-access-m6v2r\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.520413 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.520366    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpgf\" (UniqueName: \"kubernetes.io/projected/ab0a2ebd-912c-4c20-89af-cd10830e7118-kube-api-access-xkpgf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.893465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.893371    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" path="/var/lib/kubelet/pods/12ee5981-c096-4bf6-9418-7a5d5c0e1100/volumes"
Oct 30 18:56:21.894014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.893997    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642412e6-2173-408a-b9c9-1a508d72e305" path="/var/lib/kubelet/pods/642412e6-2173-408a-b9c9-1a508d72e305/volumes"
Oct 30 18:56:21.894537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:21.894521    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0a2ebd-912c-4c20-89af-cd10830e7118" path="/var/lib/kubelet/pods/ab0a2ebd-912c-4c20-89af-cd10830e7118/volumes"
Oct 30 18:56:22.483281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.483247    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"]
Oct 30 18:56:22.483977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.483961    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9f2e397e-6167-49c5-a230-ba39ba720504" containerName="revoke-keys"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.483978    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2e397e-6167-49c5-a230-ba39ba720504" containerName="revoke-keys"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484002    5065 cpu_manager.go:414] "containerMap: removing container" podUID="86e4bc45-1c84-421b-b763-3d28f2a6bdac" containerName="revoke-keys"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484007    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e4bc45-1c84-421b-b763-3d28f2a6bdac" containerName="revoke-keys"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484023    5065 cpu_manager.go:414] "containerMap: removing container" podUID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" containerName="payload-processing"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484029    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" containerName="payload-processing"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484038    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ab0a2ebd-912c-4c20-89af-cd10830e7118" containerName="payload-pre-processing"
Oct 30 18:56:22.484046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484043    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0a2ebd-912c-4c20-89af-cd10830e7118" containerName="payload-pre-processing"
Oct 30 18:56:22.484476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484189    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ee5981-c096-4bf6-9418-7a5d5c0e1100" containerName="payload-processing"
Oct 30 18:56:22.484476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484199    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0a2ebd-912c-4c20-89af-cd10830e7118" containerName="payload-pre-processing"
Oct 30 18:56:22.484476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484216    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2e397e-6167-49c5-a230-ba39ba720504" containerName="revoke-keys"
Oct 30 18:56:22.484476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.484226    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e4bc45-1c84-421b-b763-3d28f2a6bdac" containerName="revoke-keys"
Oct 30 18:56:22.489592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.489574    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.496844 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.496820    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:22.499569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.498907    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:56:22.505831 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.505801    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"]
Oct 30 18:56:22.634557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.634521    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.634738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.634699    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.634953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.634932    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4s2\" (UniqueName: \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.736172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.736076    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4s2\" (UniqueName: \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.736172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.736152    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.736404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.736223    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.736823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.736800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.738858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.738832    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.750628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.750597    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4s2\" (UniqueName: \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\") pod \"maas-api-revoke-sub-1b094b15-xnx77\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:22.808742 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:22.808719    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:23.057142 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:56:23.057102    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:56:23.104558 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:23.104533    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"]
Oct 30 18:56:23.113955 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:23.113911    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9d3be0_5437_4266_95c7_79bd7ca90a18.slice/crio-121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156 WatchSource:0}: Error finding container 121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156: Status 404 returned error can't find the container with id 121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156
Oct 30 18:56:23.471861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:23.471830    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77" event={"ID":"fe9d3be0-5437-4266-95c7-79bd7ca90a18","Type":"ContainerStarted","Data":"121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156"}
Oct 30 18:56:24.019478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.019435    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"]
Oct 30 18:56:24.024023 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.024000    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.034225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.034200    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"]
Oct 30 18:56:24.072437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.072403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.072806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.072468    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.072806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.072663    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ksrc\" (UniqueName: \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.175250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.174682    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.175250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.174751    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.175250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.174866    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ksrc\" (UniqueName: \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.183106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.176028    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.183106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.178562    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.222924 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.222818    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ksrc\" (UniqueName: \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\") pod \"maas-api-revoke-sub-e789facf-zvxf9\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.345121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.345039    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:24.488294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.488253    5065 generic.go:356] "Generic (PLEG): container finished" podID="fe9d3be0-5437-4266-95c7-79bd7ca90a18" containerID="dd8f82f35fdffd6074ef23723b1c6c289a163b3659194d156a8f0313ced1b31d" exitCode=0
Oct 30 18:56:24.488471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.488352    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77" event={"ID":"fe9d3be0-5437-4266-95c7-79bd7ca90a18","Type":"ContainerDied","Data":"dd8f82f35fdffd6074ef23723b1c6c289a163b3659194d156a8f0313ced1b31d"}
Oct 30 18:56:24.635558 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:24.635532    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fec6307_d9df_4ff6_a449_f99c1f082545.slice/crio-baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be WatchSource:0}: Error finding container baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be: Status 404 returned error can't find the container with id baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be
Oct 30 18:56:24.635840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:24.635816    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"]
Oct 30 18:56:25.503109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:25.501990    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9" event={"ID":"5fec6307-d9df-4ff6-a449-f99c1f082545","Type":"ContainerStarted","Data":"31fa70612de9dff7cd156f5009b5889f806796987df23b09f721f8598162b8e8"}
Oct 30 18:56:25.503109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:25.502062    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9" event={"ID":"5fec6307-d9df-4ff6-a449-f99c1f082545","Type":"ContainerStarted","Data":"baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be"}
Oct 30 18:56:25.530718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:25.530658    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9" podStartSLOduration=2.5306393529999998 podStartE2EDuration="2.530639353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:25.522590193 +0000 UTC m=+1405.786055161" watchObservedRunningTime="2026-10-30 18:56:25.530639353 +0000 UTC m=+1405.794104302"
Oct 30 18:56:25.904137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:25.903498    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:26.009711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.007574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\" (UniqueName: \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\") pod \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") "
Oct 30 18:56:26.009711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.007704    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\") pod \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") "
Oct 30 18:56:26.009711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.007743    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\") pod \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\" (UID: \"fe9d3be0-5437-4266-95c7-79bd7ca90a18\") "
Oct 30 18:56:26.009711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.008690    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca" pod "fe9d3be0-5437-4266-95c7-79bd7ca90a18" (UID: "fe9d3be0-5437-4266-95c7-79bd7ca90a18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:26.018482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.018450    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2" pod "fe9d3be0-5437-4266-95c7-79bd7ca90a18" (UID: "fe9d3be0-5437-4266-95c7-79bd7ca90a18"). InnerVolumeSpecName "kube-api-access-7k4s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:26.020591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.020561    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca" pod "fe9d3be0-5437-4266-95c7-79bd7ca90a18" (UID: "fe9d3be0-5437-4266-95c7-79bd7ca90a18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:26.109888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.109794    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4s2\" (UniqueName: \"kubernetes.io/projected/fe9d3be0-5437-4266-95c7-79bd7ca90a18-kube-api-access-7k4s2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.109888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.109832    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.109888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.109848    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9d3be0-5437-4266-95c7-79bd7ca90a18-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:26.517752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.517701    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"
Oct 30 18:56:26.518320 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.518291    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77" event={"ID":"fe9d3be0-5437-4266-95c7-79bd7ca90a18","Type":"ContainerDied","Data":"121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156"}
Oct 30 18:56:26.527245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.518352    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="121685497a971994716e91f52b76990cdc5f8fd249622f2b09863a7bd34b8156"
Oct 30 18:56:26.527245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.520765    5065 generic.go:356] "Generic (PLEG): container finished" podID="5fec6307-d9df-4ff6-a449-f99c1f082545" containerID="31fa70612de9dff7cd156f5009b5889f806796987df23b09f721f8598162b8e8" exitCode=0
Oct 30 18:56:26.527245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:26.520901    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9" event={"ID":"5fec6307-d9df-4ff6-a449-f99c1f082545","Type":"ContainerDied","Data":"31fa70612de9dff7cd156f5009b5889f806796987df23b09f721f8598162b8e8"}
Oct 30 18:56:27.787070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.787008    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:27.839881 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.839855    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\") pod \"5fec6307-d9df-4ff6-a449-f99c1f082545\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") "
Oct 30 18:56:27.840095 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.840062    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\" (UniqueName: \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\") pod \"5fec6307-d9df-4ff6-a449-f99c1f082545\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") "
Oct 30 18:56:27.840164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.840118    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\") pod \"5fec6307-d9df-4ff6-a449-f99c1f082545\" (UID: \"5fec6307-d9df-4ff6-a449-f99c1f082545\") "
Oct 30 18:56:27.840605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.840560    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca" pod "5fec6307-d9df-4ff6-a449-f99c1f082545" (UID: "5fec6307-d9df-4ff6-a449-f99c1f082545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:27.840954 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.840886    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:27.843151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.842723    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca" pod "5fec6307-d9df-4ff6-a449-f99c1f082545" (UID: "5fec6307-d9df-4ff6-a449-f99c1f082545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:27.843331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.843308    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc" pod "5fec6307-d9df-4ff6-a449-f99c1f082545" (UID: "5fec6307-d9df-4ff6-a449-f99c1f082545"). InnerVolumeSpecName "kube-api-access-9ksrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:27.942661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.942634    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fec6307-d9df-4ff6-a449-f99c1f082545-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:27.942790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:27.942666    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ksrc\" (UniqueName: \"kubernetes.io/projected/5fec6307-d9df-4ff6-a449-f99c1f082545-kube-api-access-9ksrc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:28.542017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:28.541992    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"
Oct 30 18:56:28.542189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:28.542029    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9" event={"ID":"5fec6307-d9df-4ff6-a449-f99c1f082545","Type":"ContainerDied","Data":"baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be"}
Oct 30 18:56:28.542189 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:28.542076    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa729c31c3341a4c123cf072a6dc2ce96e3657585e11181cef7552d6fa169be"
Oct 30 18:56:30.936580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.936546    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 18:56:30.937225 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937211    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5fec6307-d9df-4ff6-a449-f99c1f082545" containerName="revoke-keys"
Oct 30 18:56:30.937277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937227    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fec6307-d9df-4ff6-a449-f99c1f082545" containerName="revoke-keys"
Oct 30 18:56:30.937277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937246    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fe9d3be0-5437-4266-95c7-79bd7ca90a18" containerName="revoke-keys"
Oct 30 18:56:30.937277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937251    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9d3be0-5437-4266-95c7-79bd7ca90a18" containerName="revoke-keys"
Oct 30 18:56:30.937375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937355    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fec6307-d9df-4ff6-a449-f99c1f082545" containerName="revoke-keys"
Oct 30 18:56:30.937375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.937366    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9d3be0-5437-4266-95c7-79bd7ca90a18" containerName="revoke-keys"
Oct 30 18:56:30.941291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.941268    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:30.960030 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:30.959491    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 18:56:31.006296 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006261    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006322    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006389    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006609 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006515    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006609 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006568    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006607    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pd9\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006664    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006798 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006721    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.006953 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.006931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108408    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108545    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108593    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108618    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108705    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108751    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.108866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108791    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pd9\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.109040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108916    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.109040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.108981    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.109203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.109179    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.109862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.109833    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.110102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.110058    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.110259 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.110121    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.110470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.110300    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.112046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.112019    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.112613 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.112582    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.122571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.122547    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pd9\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.125405 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.125385    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.273724 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.273441    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:31.708321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.708164    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 18:56:31.717209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.717168    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:31.718129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.717560    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:31.718129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:31.717621    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:32.915322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:32.915286    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" event={"ID":"68a6722d-13ca-40ef-91a4-5730fa7f88ab","Type":"ContainerStarted","Data":"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"}
Oct 30 18:56:32.922113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:32.915932    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" event={"ID":"68a6722d-13ca-40ef-91a4-5730fa7f88ab","Type":"ContainerStarted","Data":"18faff16741de63a1d27802b32e8565935d2cae653c44a5dddcecaf63b7eee44"}
Oct 30 18:56:33.102116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.100687    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podStartSLOduration=3.100667057 podStartE2EDuration="3.100667057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:33.022268444 +0000 UTC m=+1413.285733432" watchObservedRunningTime="2026-10-30 18:56:33.100667057 +0000 UTC m=+1413.364132006"
Oct 30 18:56:33.275864 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.275834    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:33.276247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.276223    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.276420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.276397    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:33.509737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.509600    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:33.575614 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.575512    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:33.998275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:33.998172    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.288112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.281504    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.288112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.281603    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:34.580840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.580744    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:56:34.586111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.581465    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:56:34.796112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.775175    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:56:34.852172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.828205    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.984112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:34.964592    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.185168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.184115    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.366326 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.349202    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.366727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.366603    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:35.616796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.616760    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.617316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.617290    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.617675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.617656    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.750284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.750241    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 30 18:56:35.931159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.912156    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.942161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.932032    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:56:36.017648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.944494    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.130112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:35.977033    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.413003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.412969    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:36.413633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.413601    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:36.534965 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.534932    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:56:36.541218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.541176    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:56:36.546201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.542546    5065 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:56:36.574057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.573957    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:56:36.628922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.618307    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.835147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.654178    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.835147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.733401    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:56:36.835147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.821038    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:56:36.835147 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.821105    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:56:36.909280 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.909255    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:56:36.967570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:36.967537    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:56:37.106727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:37.105419    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:56:37.274994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:37.274958    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:37.275288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:37.275012    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:38.259877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.259844    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"]
Oct 30 18:56:38.264469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.264446    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.270561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.270523    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:38.270662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.270604    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:56:38.273893 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.273873    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"]
Oct 30 18:56:38.274509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.274483    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:38.274601 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.274534    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:38.331608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.330158    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.331608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.330297    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xth2\" (UniqueName: \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.331608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.330429    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.431857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.431825    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.432036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.431938    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.432036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.432027    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xth2\" (UniqueName: \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.432765 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.432736    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.434660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.434631    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.445733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.445706    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xth2\" (UniqueName: \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\") pod \"maas-api-revoke-sub-5a001c07-hz2bm\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.599573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.599287    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:38.970654 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:38.970626    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db44fb_7ce3_4b51_a6f1_14ec581f742e.slice/crio-ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee WatchSource:0}: Error finding container ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee: Status 404 returned error can't find the container with id ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee
Oct 30 18:56:38.970837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:38.970813    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"]
Oct 30 18:56:39.274502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:39.274455    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:39.275013 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:39.274512    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:39.677165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:39.677117    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm" event={"ID":"68db44fb-7ce3-4b51-a6f1-14ec581f742e","Type":"ContainerStarted","Data":"7493f49d7e94509da72fcefd5f565179e60a1e607c0c17f6333e695270d191d0"}
Oct 30 18:56:39.677365 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:39.677177    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm" event={"ID":"68db44fb-7ce3-4b51-a6f1-14ec581f742e","Type":"ContainerStarted","Data":"ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee"}
Oct 30 18:56:39.745366 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:39.745305    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm" podStartSLOduration=1.745290523 podStartE2EDuration="1.745290523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:39.742518575 +0000 UTC m=+1420.005983552" watchObservedRunningTime="2026-10-30 18:56:39.745290523 +0000 UTC m=+1420.008755472"
Oct 30 18:56:40.274518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:40.274433    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:40.274518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:40.274483    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:40.688874 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:40.688781    5065 generic.go:356] "Generic (PLEG): container finished" podID="68db44fb-7ce3-4b51-a6f1-14ec581f742e" containerID="7493f49d7e94509da72fcefd5f565179e60a1e607c0c17f6333e695270d191d0" exitCode=0
Oct 30 18:56:40.689028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:40.688872    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm" event={"ID":"68db44fb-7ce3-4b51-a6f1-14ec581f742e","Type":"ContainerDied","Data":"7493f49d7e94509da72fcefd5f565179e60a1e607c0c17f6333e695270d191d0"}
Oct 30 18:56:41.001699 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.001618    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"]
Oct 30 18:56:41.009662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.009634    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.049467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.049434    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"]
Oct 30 18:56:41.072245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.072208    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.072432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.072328    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsbx\" (UniqueName: \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.072507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.072452    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.173958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.173926    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.174202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.174107    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.174202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.174182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsbx\" (UniqueName: \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.175138 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.175077    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.177215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.177191    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.188664 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.188637    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsbx\" (UniqueName: \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\") pod \"maas-api-revoke-sub-0cc181d1-cb74h\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.274756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.274725    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 18:56:41.275188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.274771    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 18:56:41.350047 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.350011    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:41.678434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.676265    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"]
Oct 30 18:56:41.678434 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:41.678351    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303b8be0_8b94_4f04_baa3_15f80d7e8739.slice/crio-c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22 WatchSource:0}: Error finding container c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22: Status 404 returned error can't find the container with id c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22
Oct 30 18:56:41.732126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:41.731923    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h" event={"ID":"303b8be0-8b94-4f04-baa3-15f80d7e8739","Type":"ContainerStarted","Data":"c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22"}
Oct 30 18:56:42.013372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.013345    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:42.135257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.135232    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\") pod \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") "
Oct 30 18:56:42.135449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.135400    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\" (UniqueName: \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\") pod \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") "
Oct 30 18:56:42.135632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.135610    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\") pod \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\" (UID: \"68db44fb-7ce3-4b51-a6f1-14ec581f742e\") "
Oct 30 18:56:42.135993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.135970    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca" pod "68db44fb-7ce3-4b51-a6f1-14ec581f742e" (UID: "68db44fb-7ce3-4b51-a6f1-14ec581f742e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:42.136282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.136267    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.138672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.138646    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca" pod "68db44fb-7ce3-4b51-a6f1-14ec581f742e" (UID: "68db44fb-7ce3-4b51-a6f1-14ec581f742e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:42.138757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.138730    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2" pod "68db44fb-7ce3-4b51-a6f1-14ec581f742e" (UID: "68db44fb-7ce3-4b51-a6f1-14ec581f742e"). InnerVolumeSpecName "kube-api-access-2xth2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:42.237984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.237956    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xth2\" (UniqueName: \"kubernetes.io/projected/68db44fb-7ce3-4b51-a6f1-14ec581f742e-kube-api-access-2xth2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.237984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.237986    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68db44fb-7ce3-4b51-a6f1-14ec581f742e-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.352707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.352674    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:42.353191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.353115    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:42.353928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.353905    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 18:56:42.756168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.756107    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm" event={"ID":"68db44fb-7ce3-4b51-a6f1-14ec581f742e","Type":"ContainerDied","Data":"ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee"}
Oct 30 18:56:42.756168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.756168    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3b6d9f1e6fc500c5fb46c30186ef0e293c45b3337cdf0ff768b2d3876c0dee"
Oct 30 18:56:42.756443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.756240    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"
Oct 30 18:56:42.762987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:42.762496    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h" event={"ID":"303b8be0-8b94-4f04-baa3-15f80d7e8739","Type":"ContainerStarted","Data":"f513083fbf3301a0864de26cbe7db7dc9640e4ed2841ba3d197a37b9ffcf1a99"}
Oct 30 18:56:43.067554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.067495    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h" podStartSLOduration=3.067476599 podStartE2EDuration="3.067476599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:42.789797809 +0000 UTC m=+1423.053262784" watchObservedRunningTime="2026-10-30 18:56:43.067476599 +0000 UTC m=+1423.330941550"
Oct 30 18:56:43.446474 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.446390    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"]
Oct 30 18:56:43.447407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.447386    5065 cpu_manager.go:414] "containerMap: removing container" podUID="68db44fb-7ce3-4b51-a6f1-14ec581f742e" containerName="revoke-keys"
Oct 30 18:56:43.447453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.447411    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68db44fb-7ce3-4b51-a6f1-14ec581f742e" containerName="revoke-keys"
Oct 30 18:56:43.447606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.447594    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="68db44fb-7ce3-4b51-a6f1-14ec581f742e" containerName="revoke-keys"
Oct 30 18:56:43.451568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.451550    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.463858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.463827    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"]
Oct 30 18:56:43.559830 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.559799    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.560004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.559888    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdv4\" (UniqueName: \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.560004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.560000    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.661662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.661632    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.661817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.661767    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.661861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.661836    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdv4\" (UniqueName: \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.662286 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.662262    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.664244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.664219    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.675230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.675208    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdv4\" (UniqueName: \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.769570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.769543    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:43.771467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.771443    5065 generic.go:356] "Generic (PLEG): container finished" podID="303b8be0-8b94-4f04-baa3-15f80d7e8739" containerID="f513083fbf3301a0864de26cbe7db7dc9640e4ed2841ba3d197a37b9ffcf1a99" exitCode=0
Oct 30 18:56:43.771604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:43.771531    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h" event={"ID":"303b8be0-8b94-4f04-baa3-15f80d7e8739","Type":"ContainerDied","Data":"f513083fbf3301a0864de26cbe7db7dc9640e4ed2841ba3d197a37b9ffcf1a99"}
Oct 30 18:56:44.070459 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:44.070434    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb629761_6b42_4005_8497_fdb0960f7287.slice/crio-ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8 WatchSource:0}: Error finding container ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8: Status 404 returned error can't find the container with id ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8
Oct 30 18:56:44.071506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:44.071484    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"]
Oct 30 18:56:44.782324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:44.782290    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq" event={"ID":"db629761-6b42-4005-8497-fdb0960f7287","Type":"ContainerStarted","Data":"5230794bb7fac8e3e1c868f152193e127e32e475bc180fe2df4caa4fa55345c2"}
Oct 30 18:56:44.782721 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:44.782339    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq" event={"ID":"db629761-6b42-4005-8497-fdb0960f7287","Type":"ContainerStarted","Data":"ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8"}
Oct 30 18:56:44.808377 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:44.808314    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq" podStartSLOduration=1.808296468 podStartE2EDuration="1.808296468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:44.805595587 +0000 UTC m=+1425.069060989" watchObservedRunningTime="2026-10-30 18:56:44.808296468 +0000 UTC m=+1425.071761418"
Oct 30 18:56:45.026678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.026651    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:45.103989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.103962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\" (UniqueName: \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\") pod \"303b8be0-8b94-4f04-baa3-15f80d7e8739\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") "
Oct 30 18:56:45.104155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.104011    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\") pod \"303b8be0-8b94-4f04-baa3-15f80d7e8739\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") "
Oct 30 18:56:45.104222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.104167    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\") pod \"303b8be0-8b94-4f04-baa3-15f80d7e8739\" (UID: \"303b8be0-8b94-4f04-baa3-15f80d7e8739\") "
Oct 30 18:56:45.104551 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.104523    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca" pod "303b8be0-8b94-4f04-baa3-15f80d7e8739" (UID: "303b8be0-8b94-4f04-baa3-15f80d7e8739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:45.105062 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.105040    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:45.106276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.106246    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx" pod "303b8be0-8b94-4f04-baa3-15f80d7e8739" (UID: "303b8be0-8b94-4f04-baa3-15f80d7e8739"). InnerVolumeSpecName "kube-api-access-kmsbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:45.106635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.106618    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca" pod "303b8be0-8b94-4f04-baa3-15f80d7e8739" (UID: "303b8be0-8b94-4f04-baa3-15f80d7e8739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:45.205811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.205779    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmsbx\" (UniqueName: \"kubernetes.io/projected/303b8be0-8b94-4f04-baa3-15f80d7e8739-kube-api-access-kmsbx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:45.205811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.205805    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b8be0-8b94-4f04-baa3-15f80d7e8739-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:45.792895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.792856    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"
Oct 30 18:56:45.793444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.792857    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h" event={"ID":"303b8be0-8b94-4f04-baa3-15f80d7e8739","Type":"ContainerDied","Data":"c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22"}
Oct 30 18:56:45.793444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.793017    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4600e3ff670c0f67b27db08c7ee1fd14e52ea18c9a5d825bcbd5578ee8a3e22"
Oct 30 18:56:45.795160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.795134    5065 generic.go:356] "Generic (PLEG): container finished" podID="db629761-6b42-4005-8497-fdb0960f7287" containerID="5230794bb7fac8e3e1c868f152193e127e32e475bc180fe2df4caa4fa55345c2" exitCode=0
Oct 30 18:56:45.795308 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:45.795170    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq" event={"ID":"db629761-6b42-4005-8497-fdb0960f7287","Type":"ContainerDied","Data":"5230794bb7fac8e3e1c868f152193e127e32e475bc180fe2df4caa4fa55345c2"}
Oct 30 18:56:47.044122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.044082    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:47.132826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.132743    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\") pod \"db629761-6b42-4005-8497-fdb0960f7287\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") "
Oct 30 18:56:47.132826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.132789    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\") pod \"db629761-6b42-4005-8497-fdb0960f7287\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") "
Oct 30 18:56:47.133122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.133100    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca" pod "db629761-6b42-4005-8497-fdb0960f7287" (UID: "db629761-6b42-4005-8497-fdb0960f7287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:47.133188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.133144    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\" (UniqueName: \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\") pod \"db629761-6b42-4005-8497-fdb0960f7287\" (UID: \"db629761-6b42-4005-8497-fdb0960f7287\") "
Oct 30 18:56:47.133545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.133526    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db629761-6b42-4005-8497-fdb0960f7287-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.135493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.135468    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca" pod "db629761-6b42-4005-8497-fdb0960f7287" (UID: "db629761-6b42-4005-8497-fdb0960f7287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:47.135832 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.135811    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4" pod "db629761-6b42-4005-8497-fdb0960f7287" (UID: "db629761-6b42-4005-8497-fdb0960f7287"). InnerVolumeSpecName "kube-api-access-9kdv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:47.234980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.234947    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db629761-6b42-4005-8497-fdb0960f7287-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.235148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.234983    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdv4\" (UniqueName: \"kubernetes.io/projected/db629761-6b42-4005-8497-fdb0960f7287-kube-api-access-9kdv4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.816507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.816461    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"
Oct 30 18:56:47.816507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.816497    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq" event={"ID":"db629761-6b42-4005-8497-fdb0960f7287","Type":"ContainerDied","Data":"ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8"}
Oct 30 18:56:47.816729 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:47.816579    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe174c1ac4cc8b0e8f9f8f61936b55d4cd717bb6f4d6edda5a6987cbf541ea8"
Oct 30 18:56:53.058590 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:56:53.058552    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:56:54.073179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.073139    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"]
Oct 30 18:56:54.074440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074411    5065 cpu_manager.go:414] "containerMap: removing container" podUID="303b8be0-8b94-4f04-baa3-15f80d7e8739" containerName="revoke-keys"
Oct 30 18:56:54.074440 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074440    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303b8be0-8b94-4f04-baa3-15f80d7e8739" containerName="revoke-keys"
Oct 30 18:56:54.074657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074491    5065 cpu_manager.go:414] "containerMap: removing container" podUID="db629761-6b42-4005-8497-fdb0960f7287" containerName="revoke-keys"
Oct 30 18:56:54.074657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074501    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db629761-6b42-4005-8497-fdb0960f7287" containerName="revoke-keys"
Oct 30 18:56:54.074740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074700    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="303b8be0-8b94-4f04-baa3-15f80d7e8739" containerName="revoke-keys"
Oct 30 18:56:54.074740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.074726    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="db629761-6b42-4005-8497-fdb0960f7287" containerName="revoke-keys"
Oct 30 18:56:54.079465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.079444    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"]
Oct 30 18:56:54.079600 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.079549    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.084634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.084614    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:54.086578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.086540    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:56:54.178015 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.177982    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.178237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.178043    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdm6\" (UniqueName: \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.178237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.178177    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.283528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.280653    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.283528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.280754    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdm6\" (UniqueName: \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.283528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.280819    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.283528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.282416    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.296986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.296942    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.297938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.297910    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdm6\" (UniqueName: \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\") pod \"maas-api-revoke-sub-416e8ddc-xxmvj\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.400126 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.399999    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:54.697963 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:56:54.697930    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e41470b_5581_4512_a2ca_8cb33069953f.slice/crio-9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a WatchSource:0}: Error finding container 9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a: Status 404 returned error can't find the container with id 9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a
Oct 30 18:56:54.700194 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:54.700167    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"]
Oct 30 18:56:55.010493 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:55.010464    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj" event={"ID":"9e41470b-5581-4512-a2ca-8cb33069953f","Type":"ContainerStarted","Data":"f5c5d0e7b66ac66262f99c727b11af3a949f13258909d4bef0206b83c5c470a2"}
Oct 30 18:56:55.010630 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:55.010530    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj" event={"ID":"9e41470b-5581-4512-a2ca-8cb33069953f","Type":"ContainerStarted","Data":"9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a"}
Oct 30 18:56:55.039854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:55.039791    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj" podStartSLOduration=1.039773036 podStartE2EDuration="1.039773036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:55.033218768 +0000 UTC m=+1435.296683759" watchObservedRunningTime="2026-10-30 18:56:55.039773036 +0000 UTC m=+1435.303237987"
Oct 30 18:56:56.022031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:56.021987    5065 generic.go:356] "Generic (PLEG): container finished" podID="9e41470b-5581-4512-a2ca-8cb33069953f" containerID="f5c5d0e7b66ac66262f99c727b11af3a949f13258909d4bef0206b83c5c470a2" exitCode=0
Oct 30 18:56:56.022448 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:56.022071    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj" event={"ID":"9e41470b-5581-4512-a2ca-8cb33069953f","Type":"ContainerDied","Data":"f5c5d0e7b66ac66262f99c727b11af3a949f13258909d4bef0206b83c5c470a2"}
Oct 30 18:56:57.217732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.217688    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:57.410168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.399233    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\" (UniqueName: \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\") pod \"9e41470b-5581-4512-a2ca-8cb33069953f\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") "
Oct 30 18:56:57.410168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.399411    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\") pod \"9e41470b-5581-4512-a2ca-8cb33069953f\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") "
Oct 30 18:56:57.410168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.399473    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\") pod \"9e41470b-5581-4512-a2ca-8cb33069953f\" (UID: \"9e41470b-5581-4512-a2ca-8cb33069953f\") "
Oct 30 18:56:57.410168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.406440    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca" pod "9e41470b-5581-4512-a2ca-8cb33069953f" (UID: "9e41470b-5581-4512-a2ca-8cb33069953f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:57.417168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.415590    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6" pod "9e41470b-5581-4512-a2ca-8cb33069953f" (UID: "9e41470b-5581-4512-a2ca-8cb33069953f"). InnerVolumeSpecName "kube-api-access-7kdm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:57.420237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.420205    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca" pod "9e41470b-5581-4512-a2ca-8cb33069953f" (UID: "9e41470b-5581-4512-a2ca-8cb33069953f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:57.499752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.499719    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:56:57.500502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.500483    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9e41470b-5581-4512-a2ca-8cb33069953f" containerName="revoke-keys"
Oct 30 18:56:57.500502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.500503    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e41470b-5581-4512-a2ca-8cb33069953f" containerName="revoke-keys"
Oct 30 18:56:57.500732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.500716    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e41470b-5581-4512-a2ca-8cb33069953f" containerName="revoke-keys"
Oct 30 18:56:57.504736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.504716    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.512596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.512058    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:57.512596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.512108    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e41470b-5581-4512-a2ca-8cb33069953f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:57.512596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.512123    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdm6\" (UniqueName: \"kubernetes.io/projected/9e41470b-5581-4512-a2ca-8cb33069953f-kube-api-access-7kdm6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:57.519714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.517256    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-493721-openshift-default-dockercfg-nmsgp\""
Oct 30 18:56:57.527195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.526031    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:56:57.614949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.614911    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.617050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.617023    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.617338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.617321    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.617459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.617446    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.617727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.617690    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.617963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.617929    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.618034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.618012    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.618114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.618040    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.618114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.618078    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmklz\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.739784    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmklz\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.739987    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740048    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740127    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740152    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740220    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740279    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740323    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740345    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.740731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.741879    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.742474    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.743167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.742967    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.750111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.747463    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.769774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.769742    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmklz\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.774610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.774555    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.775261 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.775217    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.793744 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.793680    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\") pod \"e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") " pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:57.832537 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:57.832489    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:56:58.140239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:58.140201    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj" event={"ID":"9e41470b-5581-4512-a2ca-8cb33069953f","Type":"ContainerDied","Data":"9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a"}
Oct 30 18:56:58.144227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:58.144195    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afb8975cebfaca2392b0fdb783396150a817541a5ba01011af7363f7e9c0d6a"
Oct 30 18:56:58.144582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:58.140651    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"
Oct 30 18:56:59.030183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.030132    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:59.047387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.047347    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:59.061770 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.061675    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:56:59.228567 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.228536    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:56:59.552111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.551189    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:59.667111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.666049    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:59.675109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.672688    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:56:59.736111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.734347    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:56:59.832048 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.830546    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:59.917434 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:56:59.888141    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerStarted","Data":"5f16843a98588473e09554591a6281bc131967e40bea91e78357bce3617c5e3b"}
Oct 30 18:57:00.524454 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.524187    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.560562 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.559865    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.560920 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.560897    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:57:00.677123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.668170    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.677123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.668702    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:00.892882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.880516    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:57:00.963281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:00.956210    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:57:01.093142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.064501    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:01.093142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.064607    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:57:01.189239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.159870    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:57:01.189239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.179865    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:01.271795 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.271764    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:57:01.334111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.333829    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:01.363861 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.363830    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerStarted","Data":"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"}
Oct 30 18:57:01.469281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.469181    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podStartSLOduration=4.469143176 podStartE2EDuration="4.469143176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:01.461971906 +0000 UTC m=+1441.725436872" watchObservedRunningTime="2026-10-30 18:57:01.469143176 +0000 UTC m=+1441.732608125"
Oct 30 18:57:01.832634 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.832593    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:01.838580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:01.838548    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:02.429669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:02.429362    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:02.430432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:02.430409    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:04.263927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.263888    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"]
Oct 30 18:57:04.268586 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.268571    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.276144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.276121    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:04.276321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.276172    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:04.279934 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.279905    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"]
Oct 30 18:57:04.379077 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.379050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.379297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.379206    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhnc\" (UniqueName: \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.379297 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.379271    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.480702 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.480671    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.480883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.480801    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhnc\" (UniqueName: \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.481003 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.480976    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.481962 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.481934    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.483570 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.483548    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.495667 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.495640    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhnc\" (UniqueName: \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\") pod \"maas-api-revoke-sub-19943312-9jhkf\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.586783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.586726    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:04.884990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:04.884915    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"]
Oct 30 18:57:04.889692 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:04.889665    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2e0f72_435d_4667_a71d_ef0c4b93ce4d.slice/crio-21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad WatchSource:0}: Error finding container 21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad: Status 404 returned error can't find the container with id 21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad
Oct 30 18:57:05.496718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:05.496631    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf" event={"ID":"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d","Type":"ContainerStarted","Data":"21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad"}
Oct 30 18:57:06.507037 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:06.506953    5065 generic.go:356] "Generic (PLEG): container finished" podID="ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" containerID="b9edf8fe4e75e5289e5293927c00548a8171d026cbd3b132d9519bfaf6222ae4" exitCode=0
Oct 30 18:57:06.507467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:06.507045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf" event={"ID":"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d","Type":"ContainerDied","Data":"b9edf8fe4e75e5289e5293927c00548a8171d026cbd3b132d9519bfaf6222ae4"}
Oct 30 18:57:07.618554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.617039    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"]
Oct 30 18:57:07.625042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.625020    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"]
Oct 30 18:57:07.625206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.625127    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.692817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.692792    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:07.722279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.722254    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\" (UniqueName: \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\") pod \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") "
Oct 30 18:57:07.722410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.722336    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\") pod \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") "
Oct 30 18:57:07.722410 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.722393    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\") pod \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\" (UID: \"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d\") "
Oct 30 18:57:07.722840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.722812    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.723006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.722877    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2n6n\" (UniqueName: \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.723112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.723065    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca" pod "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" (UID: "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:07.723198 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.723111    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.723321 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.723300    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:07.725676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.725648    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca" pod "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" (UID: "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:07.725772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.725695    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc" pod "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" (UID: "ff2e0f72-435d-4667-a71d-ef0c4b93ce4d"). InnerVolumeSpecName "kube-api-access-4fhnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:07.824052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.823974    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.824239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.824153    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.824239 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.824199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2n6n\" (UniqueName: \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.824431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.824411    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fhnc\" (UniqueName: \"kubernetes.io/projected/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-kube-api-access-4fhnc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:07.824488 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.824439    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:07.824859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.824840    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.826429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.826406    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.840127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.840072    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2n6n\" (UniqueName: \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\") pod \"maas-api-revoke-sub-90a1caf0-827ll\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:07.948064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:07.948038    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:08.240375 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:08.240343    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5ef851_8a3b_46a4_a3fa_96b30f03e09a.slice/crio-3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f WatchSource:0}: Error finding container 3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f: Status 404 returned error can't find the container with id 3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f
Oct 30 18:57:08.240587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.240561    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"]
Oct 30 18:57:08.528452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.528426    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"
Oct 30 18:57:08.528452 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.528447    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf" event={"ID":"ff2e0f72-435d-4667-a71d-ef0c4b93ce4d","Type":"ContainerDied","Data":"21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad"}
Oct 30 18:57:08.528633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.528492    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a99e272c7c8e25d7d0b063d6e9811ef023f8e65c815b748777e2b327bbdfad"
Oct 30 18:57:08.530497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.530450    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll" event={"ID":"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a","Type":"ContainerStarted","Data":"f2fc65445728d5595adb8e0e05c9942aa92e7e585b593db896c128ae9738a028"}
Oct 30 18:57:08.530625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.530503    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll" event={"ID":"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a","Type":"ContainerStarted","Data":"3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f"}
Oct 30 18:57:08.553522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:08.553471    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll" podStartSLOduration=1.5534612239999999 podStartE2EDuration="1.553461224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:08.551313733 +0000 UTC m=+1448.814778683" watchObservedRunningTime="2026-10-30 18:57:08.553461224 +0000 UTC m=+1448.816926172"
Oct 30 18:57:09.308588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.308557    5065 scope.go:122] "RemoveContainer" containerID="20ee1a06ddb574c22a78370d10614133665f43cafc25b91ba6b4eefe7c6c671f"
Oct 30 18:57:09.322736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.322711    5065 scope.go:122] "RemoveContainer" containerID="4d0e3e61cb2b82f16767573f4379b2d15b45abdb40ef893f8629fade3f0f303a"
Oct 30 18:57:09.333202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.333168    5065 scope.go:122] "RemoveContainer" containerID="71a2c0cc3ce6920487fc8bdc710f5bb43bdc88d57e3821d874f477dc91bb6bbe"
Oct 30 18:57:09.343579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.343558    5065 scope.go:122] "RemoveContainer" containerID="b997927051367bb3dfa12e17002681f2297cb923c1baf3c3933666c94c06d0a1"
Oct 30 18:57:09.354620 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.354015    5065 scope.go:122] "RemoveContainer" containerID="a1e34f55a48a094c447239d1d2bf70a87688bd40c9ef45bb63b496f826fa2393"
Oct 30 18:57:09.552353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.552312    5065 generic.go:356] "Generic (PLEG): container finished" podID="9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" containerID="f2fc65445728d5595adb8e0e05c9942aa92e7e585b593db896c128ae9738a028" exitCode=0
Oct 30 18:57:09.552543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:09.552444    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll" event={"ID":"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a","Type":"ContainerDied","Data":"f2fc65445728d5595adb8e0e05c9942aa92e7e585b593db896c128ae9738a028"}
Oct 30 18:57:10.801801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.801776    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:10.889963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.889893    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\" (UniqueName: \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\") pod \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") "
Oct 30 18:57:10.890123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.890054    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\") pod \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") "
Oct 30 18:57:10.890187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.890169    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\") pod \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\" (UID: \"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a\") "
Oct 30 18:57:10.890455 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.890433    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca" pod "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" (UID: "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:10.890759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.890724    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:10.892341 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.892314    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n" pod "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" (UID: "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a"). InnerVolumeSpecName "kube-api-access-b2n6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:10.892707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.892685    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca" pod "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" (UID: "9f5ef851-8a3b-46a4-a3fa-96b30f03e09a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:10.992173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.992138    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:10.992173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:10.992172    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2n6n\" (UniqueName: \"kubernetes.io/projected/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a-kube-api-access-b2n6n\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:11.595773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:11.595746    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"
Oct 30 18:57:11.595973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:11.595746    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll" event={"ID":"9f5ef851-8a3b-46a4-a3fa-96b30f03e09a","Type":"ContainerDied","Data":"3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f"}
Oct 30 18:57:11.595973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:11.595882    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3506ba5a0531b3ab9db30b0738950fb8fbab2ece42ac35932abde5231d03044f"
Oct 30 18:57:13.537007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:13.536973    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:57:13.537482 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:13.537251    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy" containerID="cri-o://23125a2854f1b4f3a441bf701aa74c468039e98b8b0a5b841377bcb065b91d53" gracePeriod=30
Oct 30 18:57:14.107943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.107914    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"]
Oct 30 18:57:14.108602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108588    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" containerName="revoke-keys"
Oct 30 18:57:14.108645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108604    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" containerName="revoke-keys"
Oct 30 18:57:14.108645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108642    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" containerName="revoke-keys"
Oct 30 18:57:14.108719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108647    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" containerName="revoke-keys"
Oct 30 18:57:14.108772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108762    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" containerName="revoke-keys"
Oct 30 18:57:14.108811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.108778    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" containerName="revoke-keys"
Oct 30 18:57:14.112873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.112850    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.125870 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.125847    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"]
Oct 30 18:57:14.161931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.161905    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:14.162064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.162048    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:14.203695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.203664    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.203846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.203823    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.203905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.203857    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxs6g\" (UniqueName: \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.305460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.305432    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.305611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.305531    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.305611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.305554    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxs6g\" (UniqueName: \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.306423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.306396    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.308074 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.308050    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.326485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.326462    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxs6g\" (UniqueName: \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.432545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.432482    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:14.807759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:14.807732    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"]
Oct 30 18:57:14.808065 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:14.807732    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c91f44e_ab78_41b0_abcd_cc0d62d691a5.slice/crio-b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff WatchSource:0}: Error finding container b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff: Status 404 returned error can't find the container with id b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff
Oct 30 18:57:15.712676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:15.712370    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv" event={"ID":"4c91f44e-ab78-41b0-abcd-cc0d62d691a5","Type":"ContainerStarted","Data":"13f69476f08b6159029bacedf96081b96210872da5ac205dca13e454d5f0a75c"}
Oct 30 18:57:15.712676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:15.712483    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv" event={"ID":"4c91f44e-ab78-41b0-abcd-cc0d62d691a5","Type":"ContainerStarted","Data":"b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff"}
Oct 30 18:57:15.739293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:15.739236    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv" podStartSLOduration=1.739222416 podStartE2EDuration="1.739222416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:15.733728252 +0000 UTC m=+1455.997193213" watchObservedRunningTime="2026-10-30 18:57:15.739222416 +0000 UTC m=+1456.002687419"
Oct 30 18:57:16.724188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:16.724154    5065 generic.go:356] "Generic (PLEG): container finished" podID="4c91f44e-ab78-41b0-abcd-cc0d62d691a5" containerID="13f69476f08b6159029bacedf96081b96210872da5ac205dca13e454d5f0a75c" exitCode=0
Oct 30 18:57:16.724188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:16.724200    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv" event={"ID":"4c91f44e-ab78-41b0-abcd-cc0d62d691a5","Type":"ContainerDied","Data":"13f69476f08b6159029bacedf96081b96210872da5ac205dca13e454d5f0a75c"}
Oct 30 18:57:17.996381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:17.996316    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:18.068429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.068398    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\" (UniqueName: \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\") pod \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") "
Oct 30 18:57:18.068595 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.068455    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\") pod \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") "
Oct 30 18:57:18.068595 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.068511    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\") pod \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\" (UID: \"4c91f44e-ab78-41b0-abcd-cc0d62d691a5\") "
Oct 30 18:57:18.068845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.068815    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca" pod "4c91f44e-ab78-41b0-abcd-cc0d62d691a5" (UID: "4c91f44e-ab78-41b0-abcd-cc0d62d691a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:18.069552 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.069526    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:18.070843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.070820    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca" pod "4c91f44e-ab78-41b0-abcd-cc0d62d691a5" (UID: "4c91f44e-ab78-41b0-abcd-cc0d62d691a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:18.071046 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.071021    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g" pod "4c91f44e-ab78-41b0-abcd-cc0d62d691a5" (UID: "4c91f44e-ab78-41b0-abcd-cc0d62d691a5"). InnerVolumeSpecName "kube-api-access-dxs6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:18.171121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.171073    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxs6g\" (UniqueName: \"kubernetes.io/projected/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-kube-api-access-dxs6g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:18.171121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.171120    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c91f44e-ab78-41b0-abcd-cc0d62d691a5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:18.840810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.840278    5065 generic.go:356] "Generic (PLEG): container finished" podID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerID="23125a2854f1b4f3a441bf701aa74c468039e98b8b0a5b841377bcb065b91d53" exitCode=0
Oct 30 18:57:18.840810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.840570    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerDied","Data":"23125a2854f1b4f3a441bf701aa74c468039e98b8b0a5b841377bcb065b91d53"}
Oct 30 18:57:18.840810 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.840639    5065 scope.go:122] "RemoveContainer" containerID="1a374073e324e792b690251fe0a626cad3a6c0d4491a923cb2a2ad90ff85a53b"
Oct 30 18:57:18.909922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.907212    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv" event={"ID":"4c91f44e-ab78-41b0-abcd-cc0d62d691a5","Type":"ContainerDied","Data":"b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff"}
Oct 30 18:57:18.909922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.907274    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89adefcc81afb20ed569b89678a7af9e13b86272b6c53b82df7c627864e75ff"
Oct 30 18:57:18.909922 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:18.907394    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"
Oct 30 18:57:20.255175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.239486    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:20.255175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.241591    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:57:20.255175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.243800    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:57:20.347768    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c91f44e_ab78_41b0_abcd_cc0d62d691a5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.363996    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392468    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392540    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392593    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392654    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392775    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392817    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.392932    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.393016    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\") pod \"71ad147b-de5e-42f3-843e-fe309c3384fd\" (UID: \"71ad147b-de5e-42f3-843e-fe309c3384fd\") "
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.420168    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.423523    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb" event={"ID":"71ad147b-de5e-42f3-843e-fe309c3384fd","Type":"ContainerDied","Data":"a9e4cf0fbb0de6c07956cac1cf7be49e4c37cb26e18036d75dd198b62fbcf4c2"}
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.423586    5065 scope.go:122] "RemoveContainer" containerID="23125a2854f1b4f3a441bf701aa74c468039e98b8b0a5b841377bcb065b91d53"
Oct 30 18:57:20.435145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.423768    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"
Oct 30 18:57:20.476112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.458642    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:20.584974 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.584934    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:20.614323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.614274    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:20.639064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.626197    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:20.666116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.657231    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:20.709581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.708370    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:20.757358 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.757329    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.763432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.763406    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ad147b-de5e-42f3-843e-fe309c3384fd-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.763618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.763606    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.763763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.763751    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.763856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.763841    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.763950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.763939    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ad147b-de5e-42f3-843e-fe309c3384fd-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:20.879161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.878030    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "kube-api-access-vq6lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.883525    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.883581    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"]
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884610    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4c91f44e-ab78-41b0-abcd-cc0d62d691a5" containerName="revoke-keys"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884628    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c91f44e-ab78-41b0-abcd-cc0d62d691a5" containerName="revoke-keys"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884686    5065 cpu_manager.go:414] "containerMap: removing container" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884694    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884728    5065 cpu_manager.go:414] "containerMap: removing container" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884736    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884890    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:20.892120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.884904    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c91f44e-ab78-41b0-abcd-cc0d62d691a5" containerName="revoke-keys"
Oct 30 18:57:20.944150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.929101    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:20.981543 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.980834    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:21.002290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.993216    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.002290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.993281    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grk7p\" (UniqueName: \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.002290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.993442    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.002290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.993544    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.002290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:20.993563    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6lv\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-kube-api-access-vq6lv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.031190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.003410    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:21.031190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.021246    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token" pod "71ad147b-de5e-42f3-843e-fe309c3384fd" (UID: "71ad147b-de5e-42f3-843e-fe309c3384fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:21.066120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.053641    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"]
Oct 30 18:57:21.184777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.151659    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.184777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.151734    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grk7p\" (UniqueName: \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.184777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.151937    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.184777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.152248    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ad147b-de5e-42f3-843e-fe309c3384fd-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.233468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.204610    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.233468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.216353    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.233468 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.225619    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.249193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.240277    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.249193 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.240817    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:57:21.295322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.256223    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.295322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.256695    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:57:21.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.306017    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.334175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.327439    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:57:21.334175 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.327497    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.334983 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.334946    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.513166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.485828    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grk7p\" (UniqueName: \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\") pod \"maas-api-revoke-sub-f087832e-cb2n2\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:21.515294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.514654    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:57:21.629222 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.629171    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.730163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.657317    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.734112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.730677    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:57:21.830382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:21.830302    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:22.240232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.226545    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.240232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.238370    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:57:22.251520 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.251488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:57:22.292121 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.291468    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:22.318182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.318152    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"]
Oct 30 18:57:22.318710 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.318694    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fa7b20-cccjq"]
Oct 30 18:57:22.319282 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.319238    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:57:22.328041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.320501    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.355985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.355948    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.382103 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.361940    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.382103 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.371114    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:57:22.403214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.403076    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ceefc-openshift-default-76f7ffbbf7-n4zpb"]
Oct 30 18:57:22.499115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.488202    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:57:22.499456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.499432    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:57:22.499645 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.499632    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:57:22.514747 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.499783    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:57:22.517393 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.517372    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:57:22.517719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.517513    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.530109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.518136    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:22.588834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.588743    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:22.689606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.689240    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:57:22.689606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.689291    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:22.784135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:22.776671    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:23.196300 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:57:23.196189    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:57:23.201950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:23.199921    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"]
Oct 30 18:57:23.891039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:23.891010    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" path="/var/lib/kubelet/pods/71ad147b-de5e-42f3-843e-fe309c3384fd/volumes"
Oct 30 18:57:23.891533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:23.891516    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db629761-6b42-4005-8497-fdb0960f7287" path="/var/lib/kubelet/pods/db629761-6b42-4005-8497-fdb0960f7287/volumes"
Oct 30 18:57:24.023509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:24.023472    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2" event={"ID":"81a1a7ce-d87e-429a-abe3-d857118f5803","Type":"ContainerStarted","Data":"fbea7547abf46f3ac944d5be0bacaf743d2c39d8fb5335561ff66c7c3f3137e8"}
Oct 30 18:57:24.023677 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:24.023533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2" event={"ID":"81a1a7ce-d87e-429a-abe3-d857118f5803","Type":"ContainerStarted","Data":"0a77c3bddf54e2209e40c7912529c8a6898a278e6e5f5044896ac81f1b5bda71"}
Oct 30 18:57:24.057858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:24.057795    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2" podStartSLOduration=4.05778263 podStartE2EDuration="4.05778263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:24.052491868 +0000 UTC m=+1464.315956816" watchObservedRunningTime="2026-10-30 18:57:24.05778263 +0000 UTC m=+1464.321247577"
Oct 30 18:57:25.033093 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:25.033063    5065 generic.go:356] "Generic (PLEG): container finished" podID="81a1a7ce-d87e-429a-abe3-d857118f5803" containerID="fbea7547abf46f3ac944d5be0bacaf743d2c39d8fb5335561ff66c7c3f3137e8" exitCode=0
Oct 30 18:57:25.033563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:25.033151    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2" event={"ID":"81a1a7ce-d87e-429a-abe3-d857118f5803","Type":"ContainerDied","Data":"fbea7547abf46f3ac944d5be0bacaf743d2c39d8fb5335561ff66c7c3f3137e8"}
Oct 30 18:57:26.465309 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.465286    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:26.497129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.497057    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\") pod \"81a1a7ce-d87e-429a-abe3-d857118f5803\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") "
Oct 30 18:57:26.497300 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.497158    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\" (UniqueName: \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\") pod \"81a1a7ce-d87e-429a-abe3-d857118f5803\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") "
Oct 30 18:57:26.497384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.497369    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\") pod \"81a1a7ce-d87e-429a-abe3-d857118f5803\" (UID: \"81a1a7ce-d87e-429a-abe3-d857118f5803\") "
Oct 30 18:57:26.498355 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.498328    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca" pod "81a1a7ce-d87e-429a-abe3-d857118f5803" (UID: "81a1a7ce-d87e-429a-abe3-d857118f5803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:26.525882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.525840    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca" pod "81a1a7ce-d87e-429a-abe3-d857118f5803" (UID: "81a1a7ce-d87e-429a-abe3-d857118f5803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:26.528561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.528539    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p" pod "81a1a7ce-d87e-429a-abe3-d857118f5803" (UID: "81a1a7ce-d87e-429a-abe3-d857118f5803"). InnerVolumeSpecName "kube-api-access-grk7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:26.599400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.599370    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:26.599400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.599397    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grk7p\" (UniqueName: \"kubernetes.io/projected/81a1a7ce-d87e-429a-abe3-d857118f5803-kube-api-access-grk7p\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:26.599610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:26.599409    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a1a7ce-d87e-429a-abe3-d857118f5803-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:27.097624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.097590    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2" event={"ID":"81a1a7ce-d87e-429a-abe3-d857118f5803","Type":"ContainerDied","Data":"0a77c3bddf54e2209e40c7912529c8a6898a278e6e5f5044896ac81f1b5bda71"}
Oct 30 18:57:27.097884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.097867    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a77c3bddf54e2209e40c7912529c8a6898a278e6e5f5044896ac81f1b5bda71"
Oct 30 18:57:27.098039 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.098027    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"
Oct 30 18:57:27.596107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.595484    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"]
Oct 30 18:57:27.598293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.597341    5065 cpu_manager.go:414] "containerMap: removing container" podUID="81a1a7ce-d87e-429a-abe3-d857118f5803" containerName="revoke-keys"
Oct 30 18:57:27.598293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.597368    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a1a7ce-d87e-429a-abe3-d857118f5803" containerName="revoke-keys"
Oct 30 18:57:27.598293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.597831    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ad147b-de5e-42f3-843e-fe309c3384fd" containerName="istio-proxy"
Oct 30 18:57:27.598293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.597862    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a1a7ce-d87e-429a-abe3-d857118f5803" containerName="revoke-keys"
Oct 30 18:57:27.648111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.615773    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"]
Oct 30 18:57:27.648433 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.639774    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.666214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.666184    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:27.666428 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.666247    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:27.676018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.675975    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.705828 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.684343    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"]
Oct 30 18:57:27.706251 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.706073    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xw5\" (UniqueName: \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.706574 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.706532    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.724338 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.724307    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-348877-mx6nv"]
Oct 30 18:57:27.807906 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.807864    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.808119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.807930    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xw5\" (UniqueName: \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.808119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.808021    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.809035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.809003    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.815636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.815605    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.826507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.826477    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xw5\" (UniqueName: \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\") pod \"maas-api-revoke-sub-60a6e513-9tpp7\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:27.893008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:27.892942    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c91f44e-ab78-41b0-abcd-cc0d62d691a5" path="/var/lib/kubelet/pods/4c91f44e-ab78-41b0-abcd-cc0d62d691a5/volumes"
Oct 30 18:57:28.020227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:28.020201    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:28.313429 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:28.313382    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33957d35_0e51_4f51_a8c2_c8eb8c070f10.slice/crio-345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c WatchSource:0}: Error finding container 345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c: Status 404 returned error can't find the container with id 345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c
Oct 30 18:57:28.314165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:28.313996    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"]
Oct 30 18:57:29.134169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:29.130702    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7" event={"ID":"33957d35-0e51-4f51-a8c2-c8eb8c070f10","Type":"ContainerStarted","Data":"36a863e5e469c2575f19d7900436bf03f4e01f36c99cdc889d05792d9d434c71"}
Oct 30 18:57:29.134169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:29.130768    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7" event={"ID":"33957d35-0e51-4f51-a8c2-c8eb8c070f10","Type":"ContainerStarted","Data":"345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c"}
Oct 30 18:57:29.159986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:29.159920    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7" podStartSLOduration=2.159902329 podStartE2EDuration="2.159902329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:29.15614793 +0000 UTC m=+1469.419612901" watchObservedRunningTime="2026-10-30 18:57:29.159902329 +0000 UTC m=+1469.423367279"
Oct 30 18:57:30.141171 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:30.141139    5065 generic.go:356] "Generic (PLEG): container finished" podID="33957d35-0e51-4f51-a8c2-c8eb8c070f10" containerID="36a863e5e469c2575f19d7900436bf03f4e01f36c99cdc889d05792d9d434c71" exitCode=0
Oct 30 18:57:30.141561 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:30.141226    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7" event={"ID":"33957d35-0e51-4f51-a8c2-c8eb8c070f10","Type":"ContainerDied","Data":"36a863e5e469c2575f19d7900436bf03f4e01f36c99cdc889d05792d9d434c71"}
Oct 30 18:57:31.344420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.344400    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:31.472464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.472388    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\" (UniqueName: \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\") pod \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") "
Oct 30 18:57:31.472632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.472476    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\") pod \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") "
Oct 30 18:57:31.472714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.472656    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\") pod \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\" (UID: \"33957d35-0e51-4f51-a8c2-c8eb8c070f10\") "
Oct 30 18:57:31.472866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.472842    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca" pod "33957d35-0e51-4f51-a8c2-c8eb8c070f10" (UID: "33957d35-0e51-4f51-a8c2-c8eb8c070f10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:31.473096 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.473064    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:31.474647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.474618    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5" pod "33957d35-0e51-4f51-a8c2-c8eb8c070f10" (UID: "33957d35-0e51-4f51-a8c2-c8eb8c070f10"). InnerVolumeSpecName "kube-api-access-x7xw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:31.474840 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.474817    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca" pod "33957d35-0e51-4f51-a8c2-c8eb8c070f10" (UID: "33957d35-0e51-4f51-a8c2-c8eb8c070f10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:31.574979 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.574946    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33957d35-0e51-4f51-a8c2-c8eb8c070f10-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:31.575151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:31.574983    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7xw5\" (UniqueName: \"kubernetes.io/projected/33957d35-0e51-4f51-a8c2-c8eb8c070f10-kube-api-access-x7xw5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:32.161032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.160953    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7" event={"ID":"33957d35-0e51-4f51-a8c2-c8eb8c070f10","Type":"ContainerDied","Data":"345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c"}
Oct 30 18:57:32.161032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.160999    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345dbacb51ca9a6b6d6fd1e71112f83d6ee0d90890836b2b65202ad53f78de0c"
Oct 30 18:57:32.161032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.160964    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"
Oct 30 18:57:32.800483 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.800438    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"]
Oct 30 18:57:32.801672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.801648    5065 cpu_manager.go:414] "containerMap: removing container" podUID="33957d35-0e51-4f51-a8c2-c8eb8c070f10" containerName="revoke-keys"
Oct 30 18:57:32.801794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.801674    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33957d35-0e51-4f51-a8c2-c8eb8c070f10" containerName="revoke-keys"
Oct 30 18:57:32.801862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.801834    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="33957d35-0e51-4f51-a8c2-c8eb8c070f10" containerName="revoke-keys"
Oct 30 18:57:32.806648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.806625    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:32.818684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.818655    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:32.818811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.818785    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:32.902230 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.901171    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"]
Oct 30 18:57:32.906109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.905986    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:32.906109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.906079    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:32.913109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:32.906351    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lkl\" (UniqueName: \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.022110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.021934    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lkl\" (UniqueName: \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.022110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.022040    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.022400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.022130    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.040110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.035596    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.082208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.082124    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lkl\" (UniqueName: \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.085151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.085121    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8545393-jfbtd\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.163111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.162323    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:33.470465 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:33.470435    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522ff37a_3eb0_4f56_8800_17e8fa27911a.slice/crio-4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb WatchSource:0}: Error finding container 4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb: Status 404 returned error can't find the container with id 4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb
Oct 30 18:57:33.471102 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:33.471065    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"]
Oct 30 18:57:34.230496 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:34.230454    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd" event={"ID":"522ff37a-3eb0-4f56-8800-17e8fa27911a","Type":"ContainerStarted","Data":"04f4d68954e4cbea83d2d4730bf0f25cee1aabd3a36a3372f1a9bcbaec98950f"}
Oct 30 18:57:34.231717 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:34.231681    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd" event={"ID":"522ff37a-3eb0-4f56-8800-17e8fa27911a","Type":"ContainerStarted","Data":"4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb"}
Oct 30 18:57:34.304169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:34.282233    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd" podStartSLOduration=2.2822113010000002 podStartE2EDuration="2.282211301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:34.255463739 +0000 UTC m=+1474.518928727" watchObservedRunningTime="2026-10-30 18:57:34.282211301 +0000 UTC m=+1474.545676251"
Oct 30 18:57:34.839444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:34.838257    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.277116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.277064    5065 generic.go:356] "Generic (PLEG): container finished" podID="522ff37a-3eb0-4f56-8800-17e8fa27911a" containerID="04f4d68954e4cbea83d2d4730bf0f25cee1aabd3a36a3372f1a9bcbaec98950f" exitCode=0
Oct 30 18:57:35.277919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.277898    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd" event={"ID":"522ff37a-3eb0-4f56-8800-17e8fa27911a","Type":"ContainerDied","Data":"04f4d68954e4cbea83d2d4730bf0f25cee1aabd3a36a3372f1a9bcbaec98950f"}
Oct 30 18:57:35.384240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.384195    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.384845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.384814    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.507008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.506440    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.512766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.512727    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.513957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.513928    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.635176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.629912    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.720137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.719312    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.885111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:35.862318    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.031422 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.031392    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:36.220603 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.220567    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:57:36.231611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.230540    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.388278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.388177    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.502165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.418066    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:57:36.754166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.613338    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.754166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.649691    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:57:36.803284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.803254    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:57:36.803564 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:36.803548    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:37.174111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.165692    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.411113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.407356    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.442164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.434172    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:57:37.780339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.779183    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.791817 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.791779    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.802248 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.802212    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.824238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.824194    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:37.884119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:37.882058    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:57:38.121502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.121411    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:57:38.121762 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.121734    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:57:38.185971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.185939    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:57:38.639781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.639751    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:38.728185 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.728146    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:57:38.729534 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.729506    5065 cpu_manager.go:414] "containerMap: removing container" podUID="522ff37a-3eb0-4f56-8800-17e8fa27911a" containerName="revoke-keys"
Oct 30 18:57:38.729766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.729751    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522ff37a-3eb0-4f56-8800-17e8fa27911a" containerName="revoke-keys"
Oct 30 18:57:38.730154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.730138    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="522ff37a-3eb0-4f56-8800-17e8fa27911a" containerName="revoke-keys"
Oct 30 18:57:38.752909 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.752864    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.758970 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.758894    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:57:38.760324 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.760294    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fc28ee-openshift-default-dockercfg-pm7z8\""
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.855601    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\") pod \"522ff37a-3eb0-4f56-8800-17e8fa27911a\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") "
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.865216    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\") pod \"522ff37a-3eb0-4f56-8800-17e8fa27911a\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") "
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.865271    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\" (UniqueName: \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\") pod \"522ff37a-3eb0-4f56-8800-17e8fa27911a\" (UID: \"522ff37a-3eb0-4f56-8800-17e8fa27911a\") "
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.865771    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.865833    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxz7j\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.865998    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866035    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866106    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866146    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866184    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866230    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.868387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.866263    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.875456 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.875417    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca" pod "522ff37a-3eb0-4f56-8800-17e8fa27911a" (UID: "522ff37a-3eb0-4f56-8800-17e8fa27911a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:38.881009 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.880970    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl" pod "522ff37a-3eb0-4f56-8800-17e8fa27911a" (UID: "522ff37a-3eb0-4f56-8800-17e8fa27911a"). InnerVolumeSpecName "kube-api-access-r9lkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:38.933526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.933237    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca" pod "522ff37a-3eb0-4f56-8800-17e8fa27911a" (UID: "522ff37a-3eb0-4f56-8800-17e8fa27911a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:38.967425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967387    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.967616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967590    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz7j\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.967868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967846    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.967868 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967859    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967894    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968018 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.967954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968017    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968062    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968139    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968163    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968209    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968334    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968441    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:38.968700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968655    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.968700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968680    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ff37a-3eb0-4f56-8800-17e8fa27911a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:38.968945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968724    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lkl\" (UniqueName: \"kubernetes.io/projected/522ff37a-3eb0-4f56-8800-17e8fa27911a-kube-api-access-r9lkl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:38.968945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.968735    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.969878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.969853    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd" event={"ID":"522ff37a-3eb0-4f56-8800-17e8fa27911a","Type":"ContainerDied","Data":"4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb"}
Oct 30 18:57:38.969996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.969899    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed9f2271f014bb0777c86f735b523d3889f4d19a9b26f5521e9eba4b600c3bb"
Oct 30 18:57:38.969996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.969956    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"
Oct 30 18:57:38.971053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.971028    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:38.971254 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:38.971233    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:39.050834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.050802    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:39.099283 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.098176    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz7j\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\") pod \"e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") " pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:39.121258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.121220    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:39.461786 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:39.461752    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf02edf3_6113_4616_9056_c099add9960e.slice/crio-30379d26c6ded16d20dcc8f1d71492b62216313b947e36667f72d7d2f0413f08 WatchSource:0}: Error finding container 30379d26c6ded16d20dcc8f1d71492b62216313b947e36667f72d7d2f0413f08: Status 404 returned error can't find the container with id 30379d26c6ded16d20dcc8f1d71492b62216313b947e36667f72d7d2f0413f08
Oct 30 18:57:39.462472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.462445    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:57:39.464441 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.464403    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:39.464527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.464486    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:39.464581 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:39.464532    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:40.036649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:40.036612    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerStarted","Data":"cc67f8059c2b10c955bccc72ac15758c221de7774752bf59be9e03c4dfe162cf"}
Oct 30 18:57:40.037242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:40.037215    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerStarted","Data":"30379d26c6ded16d20dcc8f1d71492b62216313b947e36667f72d7d2f0413f08"}
Oct 30 18:57:40.069040 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:40.068975    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podStartSLOduration=2.068963139 podStartE2EDuration="2.068963139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:40.06782763 +0000 UTC m=+1480.331292581" watchObservedRunningTime="2026-10-30 18:57:40.068963139 +0000 UTC m=+1480.332428087"
Oct 30 18:57:40.122531 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:40.122488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:40.128373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:40.128344    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:41.063401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.062428    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:41.067897 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.067857    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:57:41.670327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.670297    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"]
Oct 30 18:57:41.674460 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.674442    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.701112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.698944    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:41.701112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.699059    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:41.732349 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.732208    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"]
Oct 30 18:57:41.774167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.774141    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.774294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.774257    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qswr\" (UniqueName: \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.774343 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.774302    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.875451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.875407    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.875626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.875525    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qswr\" (UniqueName: \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.875626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.875566    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.876228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.876207    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.877972 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.877954    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:41.919243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:41.919215    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qswr\" (UniqueName: \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\") pod \"maas-api-revoke-sub-5f3b2ca9-52jnw\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:42.059671 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:42.059383    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:42.405544 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:42.405503    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c68a2f_7584_44ef_80bc_d9c461c71905.slice/crio-a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa WatchSource:0}: Error finding container a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa: Status 404 returned error can't find the container with id a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa
Oct 30 18:57:42.408247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:42.408221    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"]
Oct 30 18:57:43.056453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.056423    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"]
Oct 30 18:57:43.060741 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.060718    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccc85d1-t6kgz"]
Oct 30 18:57:43.142111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.141443    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw" event={"ID":"82c68a2f-7584-44ef-80bc-d9c461c71905","Type":"ContainerStarted","Data":"d5ef7e66c787e8a0480ee48ed558fe369739597a84b335f645d288e4c9ac95c1"}
Oct 30 18:57:43.142111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.141506    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw" event={"ID":"82c68a2f-7584-44ef-80bc-d9c461c71905","Type":"ContainerStarted","Data":"a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa"}
Oct 30 18:57:43.192293 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.192243    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw" podStartSLOduration=2.192231424 podStartE2EDuration="2.192231424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:43.18866949 +0000 UTC m=+1483.452134439" watchObservedRunningTime="2026-10-30 18:57:43.192231424 +0000 UTC m=+1483.455696377"
Oct 30 18:57:43.894516 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:43.894485    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d509ea9f-1c78-41da-a66e-5013cda08ad5" path="/var/lib/kubelet/pods/d509ea9f-1c78-41da-a66e-5013cda08ad5/volumes"
Oct 30 18:57:44.205153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:44.203206    5065 generic.go:356] "Generic (PLEG): container finished" podID="82c68a2f-7584-44ef-80bc-d9c461c71905" containerID="d5ef7e66c787e8a0480ee48ed558fe369739597a84b335f645d288e4c9ac95c1" exitCode=0
Oct 30 18:57:44.205153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:44.203303    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw" event={"ID":"82c68a2f-7584-44ef-80bc-d9c461c71905","Type":"ContainerDied","Data":"d5ef7e66c787e8a0480ee48ed558fe369739597a84b335f645d288e4c9ac95c1"}
Oct 30 18:57:45.506053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.505966    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:45.661207 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.661171    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\" (UniqueName: \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\") pod \"82c68a2f-7584-44ef-80bc-d9c461c71905\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") "
Oct 30 18:57:45.661429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.661329    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\") pod \"82c68a2f-7584-44ef-80bc-d9c461c71905\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") "
Oct 30 18:57:45.661626 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.661602    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\") pod \"82c68a2f-7584-44ef-80bc-d9c461c71905\" (UID: \"82c68a2f-7584-44ef-80bc-d9c461c71905\") "
Oct 30 18:57:45.661942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.661914    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca" pod "82c68a2f-7584-44ef-80bc-d9c461c71905" (UID: "82c68a2f-7584-44ef-80bc-d9c461c71905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:45.662572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.662545    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:45.663679 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.663648    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr" pod "82c68a2f-7584-44ef-80bc-d9c461c71905" (UID: "82c68a2f-7584-44ef-80bc-d9c461c71905"). InnerVolumeSpecName "kube-api-access-2qswr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:45.664155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.664127    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca" pod "82c68a2f-7584-44ef-80bc-d9c461c71905" (UID: "82c68a2f-7584-44ef-80bc-d9c461c71905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:45.764008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.763979    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c68a2f-7584-44ef-80bc-d9c461c71905-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:45.764008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:45.764006    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qswr\" (UniqueName: \"kubernetes.io/projected/82c68a2f-7584-44ef-80bc-d9c461c71905-kube-api-access-2qswr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:46.227052 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:46.226965    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"
Oct 30 18:57:46.227288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:46.226965    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw" event={"ID":"82c68a2f-7584-44ef-80bc-d9c461c71905","Type":"ContainerDied","Data":"a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa"}
Oct 30 18:57:46.227288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:46.227145    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6318a449adbcccb3602d9838a4edad09870534366b59f0737f7b22b3eef6cfa"
Oct 30 18:57:47.238643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.238550    5065 generic.go:356] "Generic (PLEG): container finished" podID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerID="834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424" exitCode=0
Oct 30 18:57:47.238643 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.238619    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerDied","Data":"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"}
Oct 30 18:57:47.239139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.238996    5065 scope.go:122] "RemoveContainer" containerID="834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"
Oct 30 18:57:47.240655 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.240626    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:47.240743 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.240699    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:47.240792 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.240739    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:57:47.428927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:47.428900    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:48.251155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:48.251120    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerStarted","Data":"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"}
Oct 30 18:57:48.840191 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:48.840154    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:48.852203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:48.852169    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 18:57:48.852526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:48.852491    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 18:57:49.769173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.769140    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"]
Oct 30 18:57:49.769911 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.769894    5065 cpu_manager.go:414] "containerMap: removing container" podUID="82c68a2f-7584-44ef-80bc-d9c461c71905" containerName="revoke-keys"
Oct 30 18:57:49.769957 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.769912    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c68a2f-7584-44ef-80bc-d9c461c71905" containerName="revoke-keys"
Oct 30 18:57:49.770043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.770027    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c68a2f-7584-44ef-80bc-d9c461c71905" containerName="revoke-keys"
Oct 30 18:57:49.773887 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.773871    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:49.803829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.803563    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:49.803829 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.803626    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:49.804161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.804076    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"]
Oct 30 18:57:49.836327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.836298    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 18:57:49.836501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.836347    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 18:57:49.950265 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.950235    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:49.950427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.950279    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:49.950427 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:49.950370    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jz4\" (UniqueName: \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.052029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.051943    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.052029 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.051993    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.052257 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.052045    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jz4\" (UniqueName: \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.052739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.052710    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.054665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.054642    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.068859 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.068834    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jz4\" (UniqueName: \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\") pod \"maas-api-revoke-sub-4b05d02a-dffxl\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.126527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.126488    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:50.454256 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:50.454220    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804eaaa1_5701_471b_a871_2f59fb19a692.slice/crio-c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f WatchSource:0}: Error finding container c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f: Status 404 returned error can't find the container with id c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f
Oct 30 18:57:50.455727 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.455694    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"]
Oct 30 18:57:50.834114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.833878    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 18:57:50.834114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:50.833936    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 18:57:51.293111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:51.289972    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl" event={"ID":"804eaaa1-5701-471b-a871-2f59fb19a692","Type":"ContainerStarted","Data":"cead6db4143b8aa0d5c8b28ddbe60229a5216086550955cc6ee756a62820c2a4"}
Oct 30 18:57:51.293111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:51.290042    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl" event={"ID":"804eaaa1-5701-471b-a871-2f59fb19a692","Type":"ContainerStarted","Data":"c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f"}
Oct 30 18:57:51.326020 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:51.325970    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl" podStartSLOduration=2.325958027 podStartE2EDuration="2.325958027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:51.320396885 +0000 UTC m=+1491.583861834" watchObservedRunningTime="2026-10-30 18:57:51.325958027 +0000 UTC m=+1491.589422974"
Oct 30 18:57:51.833528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:51.833451    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 18:57:51.833723 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:51.833592    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 18:57:52.320306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:52.320266    5065 generic.go:356] "Generic (PLEG): container finished" podID="804eaaa1-5701-471b-a871-2f59fb19a692" containerID="cead6db4143b8aa0d5c8b28ddbe60229a5216086550955cc6ee756a62820c2a4" exitCode=0
Oct 30 18:57:52.321698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:52.321434    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl" event={"ID":"804eaaa1-5701-471b-a871-2f59fb19a692","Type":"ContainerDied","Data":"cead6db4143b8aa0d5c8b28ddbe60229a5216086550955cc6ee756a62820c2a4"}
Oct 30 18:57:52.835590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:52.835051    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 18:57:52.835590 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:52.835148    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 18:57:53.058663 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:57:53.058610    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:57:53.566666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.566632    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:53.629017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.628913    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\") pod \"804eaaa1-5701-471b-a871-2f59fb19a692\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") "
Oct 30 18:57:53.629017 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.628991    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\") pod \"804eaaa1-5701-471b-a871-2f59fb19a692\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") "
Oct 30 18:57:53.629326 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.629075    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\" (UniqueName: \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\") pod \"804eaaa1-5701-471b-a871-2f59fb19a692\" (UID: \"804eaaa1-5701-471b-a871-2f59fb19a692\") "
Oct 30 18:57:53.630523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.630471    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca" pod "804eaaa1-5701-471b-a871-2f59fb19a692" (UID: "804eaaa1-5701-471b-a871-2f59fb19a692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:53.632731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.632697    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4" pod "804eaaa1-5701-471b-a871-2f59fb19a692" (UID: "804eaaa1-5701-471b-a871-2f59fb19a692"). InnerVolumeSpecName "kube-api-access-h8jz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:53.633540 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.633510    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca" pod "804eaaa1-5701-471b-a871-2f59fb19a692" (UID: "804eaaa1-5701-471b-a871-2f59fb19a692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:53.730995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.730961    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:53.730995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.730996    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804eaaa1-5701-471b-a871-2f59fb19a692-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:53.731234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.731011    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jz4\" (UniqueName: \"kubernetes.io/projected/804eaaa1-5701-471b-a871-2f59fb19a692-kube-api-access-h8jz4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:53.837224 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.837200    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:53.837591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.837563    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:53.838156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:53.838136    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:57:54.344360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:54.344335    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"
Oct 30 18:57:54.344529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:54.344371    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl" event={"ID":"804eaaa1-5701-471b-a871-2f59fb19a692","Type":"ContainerDied","Data":"c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f"}
Oct 30 18:57:54.344529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:54.344422    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3cdb75f187439e9ab66debfc82df9f6caf4bedca080a1f3704a6dbf77a3316f"
Oct 30 18:57:56.317976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.317932    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"]
Oct 30 18:57:56.319035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.319011    5065 cpu_manager.go:414] "containerMap: removing container" podUID="804eaaa1-5701-471b-a871-2f59fb19a692" containerName="revoke-keys"
Oct 30 18:57:56.319035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.319036    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804eaaa1-5701-471b-a871-2f59fb19a692" containerName="revoke-keys"
Oct 30 18:57:56.319272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.319257    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="804eaaa1-5701-471b-a871-2f59fb19a692" containerName="revoke-keys"
Oct 30 18:57:56.324099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.324069    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.331985 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.331961    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:56.334472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.333655    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:57:56.334736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.334717    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"]
Oct 30 18:57:56.483587 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.483538    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv88f\" (UniqueName: \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.483805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.483724    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.483955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.483933    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.585467 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.585388    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.585649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.585525    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.585649 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.585587    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv88f\" (UniqueName: \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.586299 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.586269    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.588274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.588249    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.604137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.604113    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv88f\" (UniqueName: \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\") pod \"maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.657446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.657419    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:56.969830 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:57:56.969796    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0f4533_6fb1_4393_bfa9_69d00325e0be.slice/crio-f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e WatchSource:0}: Error finding container f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e: Status 404 returned error can't find the container with id f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e
Oct 30 18:57:56.970989 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:56.970968    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"]
Oct 30 18:57:57.450903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:57.450822    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69" event={"ID":"eb0f4533-6fb1-4393-bfa9-69d00325e0be","Type":"ContainerStarted","Data":"f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e"}
Oct 30 18:57:58.669794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:58.669758    5065 generic.go:356] "Generic (PLEG): container finished" podID="eb0f4533-6fb1-4393-bfa9-69d00325e0be" containerID="442a524ff4cbdf746f38ac778460cf242a98f7533e50c7df5b69b1f070c5d74a" exitCode=0
Oct 30 18:57:58.671144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:58.670950    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69" event={"ID":"eb0f4533-6fb1-4393-bfa9-69d00325e0be","Type":"ContainerDied","Data":"442a524ff4cbdf746f38ac778460cf242a98f7533e50c7df5b69b1f070c5d74a"}
Oct 30 18:57:59.859314 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.859289    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:57:59.915571 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.915532    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\" (UniqueName: \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\") pod \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") "
Oct 30 18:57:59.915768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.915624    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\") pod \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") "
Oct 30 18:57:59.915768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.915683    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\") pod \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\" (UID: \"eb0f4533-6fb1-4393-bfa9-69d00325e0be\") "
Oct 30 18:57:59.917202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.917162    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca" pod "eb0f4533-6fb1-4393-bfa9-69d00325e0be" (UID: "eb0f4533-6fb1-4393-bfa9-69d00325e0be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:59.928529 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.928500    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f" pod "eb0f4533-6fb1-4393-bfa9-69d00325e0be" (UID: "eb0f4533-6fb1-4393-bfa9-69d00325e0be"). InnerVolumeSpecName "kube-api-access-bv88f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:59.929359 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:57:59.929312    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca" pod "eb0f4533-6fb1-4393-bfa9-69d00325e0be" (UID: "eb0f4533-6fb1-4393-bfa9-69d00325e0be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:00.018608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.018549    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv88f\" (UniqueName: \"kubernetes.io/projected/eb0f4533-6fb1-4393-bfa9-69d00325e0be-kube-api-access-bv88f\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:00.018608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.018594    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:00.018608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.018610    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0f4533-6fb1-4393-bfa9-69d00325e0be-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:00.418712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.418683    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"]
Oct 30 18:58:00.419404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.419382    5065 cpu_manager.go:414] "containerMap: removing container" podUID="eb0f4533-6fb1-4393-bfa9-69d00325e0be" containerName="revoke-keys"
Oct 30 18:58:00.419404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.419399    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0f4533-6fb1-4393-bfa9-69d00325e0be" containerName="revoke-keys"
Oct 30 18:58:00.419580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.419555    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0f4533-6fb1-4393-bfa9-69d00325e0be" containerName="revoke-keys"
Oct 30 18:58:00.423473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.423455    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.435538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.435516    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"]
Oct 30 18:58:00.525014 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.524983    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.525173 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.525126    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.525249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.525233    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8696g\" (UniqueName: \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.626288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.626264    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8696g\" (UniqueName: \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.626431 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.626378    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.626475 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.626430    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.627377 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.627304    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.629367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.629344    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.639980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.639960    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8696g\" (UniqueName: \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\") pod \"maas-api-revoke-sub-81cf5631-cf5wg\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:00.690902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.690850    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"
Oct 30 18:58:00.690993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.690848    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69" event={"ID":"eb0f4533-6fb1-4393-bfa9-69d00325e0be","Type":"ContainerDied","Data":"f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e"}
Oct 30 18:58:00.690993 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.690971    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7ea30d9722300c721d547a8729ff40d3fbe496b8d30a711948208a86ea4962e"
Oct 30 18:58:00.739291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:00.739274    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:01.029138 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:01.029060    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077501db_9736_421f_8262_b43ca22469cc.slice/crio-ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3 WatchSource:0}: Error finding container ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3: Status 404 returned error can't find the container with id ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3
Oct 30 18:58:01.031823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:01.031794    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"]
Oct 30 18:58:01.701306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:01.701222    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg" event={"ID":"077501db-9736-421f-8262-b43ca22469cc","Type":"ContainerStarted","Data":"5c0b7f8591cdf4dff23fc1b7d8373d20c58906411116f6e9f56e462f58ced2a9"}
Oct 30 18:58:01.701306 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:01.701293    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg" event={"ID":"077501db-9736-421f-8262-b43ca22469cc","Type":"ContainerStarted","Data":"ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3"}
Oct 30 18:58:01.731577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:01.731521    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg" podStartSLOduration=1.731502942 podStartE2EDuration="1.731502942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:01.724624954 +0000 UTC m=+1501.988089928" watchObservedRunningTime="2026-10-30 18:58:01.731502942 +0000 UTC m=+1501.994967891"
Oct 30 18:58:02.010636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.010611    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"]
Oct 30 18:58:02.015061 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.015045    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.039114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.027430    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"]
Oct 30 18:58:02.230154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.230009    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.230154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.230080    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftqq\" (UniqueName: \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.230348 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.230233    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.331466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.331431    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.331751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.331721    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.331845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.331777    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftqq\" (UniqueName: \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.332425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.332403    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.334027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.334003    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.346228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.346207    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftqq\" (UniqueName: \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\") pod \"maas-api-revoke-sub-e7e8883a-bfvfb\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.447857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.447825    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:02.889568 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.883569    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"]
Oct 30 18:58:02.905109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.897914    5065 generic.go:356] "Generic (PLEG): container finished" podID="077501db-9736-421f-8262-b43ca22469cc" containerID="5c0b7f8591cdf4dff23fc1b7d8373d20c58906411116f6e9f56e462f58ced2a9" exitCode=0
Oct 30 18:58:02.905109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:02.898357    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg" event={"ID":"077501db-9736-421f-8262-b43ca22469cc","Type":"ContainerDied","Data":"5c0b7f8591cdf4dff23fc1b7d8373d20c58906411116f6e9f56e462f58ced2a9"}
Oct 30 18:58:03.401356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.401323    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:58:03.412611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.412588    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:58:03.467884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.467864    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 18:58:03.476404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.476387    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 18:58:03.925991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.925955    5065 generic.go:356] "Generic (PLEG): container finished" podID="59d7b64c-f2c9-480e-9471-35a7c816453a" containerID="9eab4ba578651283f160ad5b6332c93ae6c26254ae881bbbdeb24ce76979aff4" exitCode=0
Oct 30 18:58:03.926247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.926217    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb" event={"ID":"59d7b64c-f2c9-480e-9471-35a7c816453a","Type":"ContainerDied","Data":"9eab4ba578651283f160ad5b6332c93ae6c26254ae881bbbdeb24ce76979aff4"}
Oct 30 18:58:03.926429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:03.926283    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb" event={"ID":"59d7b64c-f2c9-480e-9471-35a7c816453a","Type":"ContainerStarted","Data":"9c7521f784520096a741a84bd7cd5d2007f23a820f6b562301a3a3fb7b7c1d9f"}
Oct 30 18:58:04.146969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.146909    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:04.290127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.283841    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\") pod \"077501db-9736-421f-8262-b43ca22469cc\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") "
Oct 30 18:58:04.290127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.283933    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\" (UniqueName: \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\") pod \"077501db-9736-421f-8262-b43ca22469cc\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") "
Oct 30 18:58:04.290127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.284024    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\") pod \"077501db-9736-421f-8262-b43ca22469cc\" (UID: \"077501db-9736-421f-8262-b43ca22469cc\") "
Oct 30 18:58:04.290127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.289824    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca" pod "077501db-9736-421f-8262-b43ca22469cc" (UID: "077501db-9736-421f-8262-b43ca22469cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:04.311129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.296778    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca" pod "077501db-9736-421f-8262-b43ca22469cc" (UID: "077501db-9736-421f-8262-b43ca22469cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:04.311129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.297226    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g" pod "077501db-9736-421f-8262-b43ca22469cc" (UID: "077501db-9736-421f-8262-b43ca22469cc"). InnerVolumeSpecName "kube-api-access-8696g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:04.386219 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.386185    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077501db-9736-421f-8262-b43ca22469cc-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:04.386403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.386255    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077501db-9736-421f-8262-b43ca22469cc-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:04.386403 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.386268    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8696g\" (UniqueName: \"kubernetes.io/projected/077501db-9736-421f-8262-b43ca22469cc-kube-api-access-8696g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:04.937767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.937740    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"
Oct 30 18:58:04.938183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.937780    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg" event={"ID":"077501db-9736-421f-8262-b43ca22469cc","Type":"ContainerDied","Data":"ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3"}
Oct 30 18:58:04.938183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:04.937826    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea93b4050c0d26054ea1eecf5043ef0534982f052ade6c43affa10539f631eb3"
Oct 30 18:58:05.216674 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.216612    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:05.403510 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.403474    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\" (UniqueName: \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\") pod \"59d7b64c-f2c9-480e-9471-35a7c816453a\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") "
Oct 30 18:58:05.403682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.403635    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\") pod \"59d7b64c-f2c9-480e-9471-35a7c816453a\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") "
Oct 30 18:58:05.403746 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.403699    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\") pod \"59d7b64c-f2c9-480e-9471-35a7c816453a\" (UID: \"59d7b64c-f2c9-480e-9471-35a7c816453a\") "
Oct 30 18:58:05.404188 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.404154    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca" pod "59d7b64c-f2c9-480e-9471-35a7c816453a" (UID: "59d7b64c-f2c9-480e-9471-35a7c816453a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:05.404550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.404527    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:05.405857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.405828    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq" pod "59d7b64c-f2c9-480e-9471-35a7c816453a" (UID: "59d7b64c-f2c9-480e-9471-35a7c816453a"). InnerVolumeSpecName "kube-api-access-bftqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:05.406011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.405893    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca" pod "59d7b64c-f2c9-480e-9471-35a7c816453a" (UID: "59d7b64c-f2c9-480e-9471-35a7c816453a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:05.505625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.505563    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftqq\" (UniqueName: \"kubernetes.io/projected/59d7b64c-f2c9-480e-9471-35a7c816453a-kube-api-access-bftqq\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:05.505625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.505587    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d7b64c-f2c9-480e-9471-35a7c816453a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:05.946773 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.946747    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"
Oct 30 18:58:05.947146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.946778    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb" event={"ID":"59d7b64c-f2c9-480e-9471-35a7c816453a","Type":"ContainerDied","Data":"9c7521f784520096a741a84bd7cd5d2007f23a820f6b562301a3a3fb7b7c1d9f"}
Oct 30 18:58:05.947146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:05.946822    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7521f784520096a741a84bd7cd5d2007f23a820f6b562301a3a3fb7b7c1d9f"
Oct 30 18:58:09.429712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.429677    5065 scope.go:122] "RemoveContainer" containerID="a487aab2eafae39991567451646fe61124b3398abe75932f9a60dc6be4fd1077"
Oct 30 18:58:09.533975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.533942    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:58:09.534757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.534736    5065 cpu_manager.go:414] "containerMap: removing container" podUID="59d7b64c-f2c9-480e-9471-35a7c816453a" containerName="revoke-keys"
Oct 30 18:58:09.534862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.534758    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d7b64c-f2c9-480e-9471-35a7c816453a" containerName="revoke-keys"
Oct 30 18:58:09.534862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.534809    5065 cpu_manager.go:414] "containerMap: removing container" podUID="077501db-9736-421f-8262-b43ca22469cc" containerName="revoke-keys"
Oct 30 18:58:09.534862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.534817    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077501db-9736-421f-8262-b43ca22469cc" containerName="revoke-keys"
Oct 30 18:58:09.535065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.534982    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="077501db-9736-421f-8262-b43ca22469cc" containerName="revoke-keys"
Oct 30 18:58:09.535065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.535008    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d7b64c-f2c9-480e-9471-35a7c816453a" containerName="revoke-keys"
Oct 30 18:58:09.540186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.540167    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.557232 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.557210    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:58:09.565258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565226    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565339    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565399    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565466    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565554 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565535    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565580    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565629    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgqm\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565713 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565680    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.565878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.565768    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.573177 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.573066    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bcb5af-openshift-default-dockercfg-zkznr\""
Oct 30 18:58:09.667582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667546    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.667764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667698    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.667764 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667756    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.667904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667787    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.667904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667859    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667935    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667979    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.667988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.668019    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgqm\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.668062    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.668752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.668721    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.669182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.669157    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.669487 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.669464    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.669639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.669613    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.670506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.670485    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.671285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.671166    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.682447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.682388    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.688060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.688039    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgqm\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\") pod \"e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") " pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:09.872285 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:09.872003    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:10.144317 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.144287    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"]
Oct 30 18:58:10.153592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.153481    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-50b19d-chm69"]
Oct 30 18:58:10.398887 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:10.398860    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cac2b4_a66a_4a5e_b26a_447af4bfb062.slice/crio-993ed86f8f94c9eb1d2b87de154e35f9697bceb58512c1f7dce87e67afb7adb9 WatchSource:0}: Error finding container 993ed86f8f94c9eb1d2b87de154e35f9697bceb58512c1f7dce87e67afb7adb9: Status 404 returned error can't find the container with id 993ed86f8f94c9eb1d2b87de154e35f9697bceb58512c1f7dce87e67afb7adb9
Oct 30 18:58:10.399153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.399135    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:58:10.402478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.402444    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:10.402569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.402511    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:10.402569 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.402541    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:10.708685 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.708600    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"]
Oct 30 18:58:10.714517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.714493    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.728195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.728112    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"]
Oct 30 18:58:10.745573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.745546    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:10.745722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.745619    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:10.842334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.842305    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.842512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.842457    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.842629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.842610    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psh7w\" (UniqueName: \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.944517 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.944474    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.944756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.944556    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.944756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.944643    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psh7w\" (UniqueName: \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.945478 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.945445    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.947169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.947134    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:10.960781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:10.960731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psh7w\" (UniqueName: \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\") pod \"maas-api-revoke-sub-2c7df416-qg8wv\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:11.040777 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.040747    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:11.354612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.353623    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerStarted","Data":"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"}
Oct 30 18:58:11.354612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.353692    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerStarted","Data":"993ed86f8f94c9eb1d2b87de154e35f9697bceb58512c1f7dce87e67afb7adb9"}
Oct 30 18:58:11.403501 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.403445    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podStartSLOduration=2.403426932 podStartE2EDuration="2.403426932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:11.399767681 +0000 UTC m=+1511.663232630" watchObservedRunningTime="2026-10-30 18:58:11.403426932 +0000 UTC m=+1511.666891881"
Oct 30 18:58:11.707289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.707211    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"]
Oct 30 18:58:11.714983 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:11.714956    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eb5945_f438_4b9f_9f18_4fef919850bf.slice/crio-0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e WatchSource:0}: Error finding container 0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e: Status 404 returned error can't find the container with id 0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e
Oct 30 18:58:11.875116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.874559    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:11.944111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:11.939351    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0f4533-6fb1-4393-bfa9-69d00325e0be" path="/var/lib/kubelet/pods/eb0f4533-6fb1-4393-bfa9-69d00325e0be/volumes"
Oct 30 18:58:12.043423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.043390    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:12.429139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.428793    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv" event={"ID":"28eb5945-f438-4b9f-9f18-4fef919850bf","Type":"ContainerStarted","Data":"2d7cf662dbdb21738ed5e091629bea4e478c286140c18a9d1b7f9760b1fe8860"}
Oct 30 18:58:12.429139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.428859    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv" event={"ID":"28eb5945-f438-4b9f-9f18-4fef919850bf","Type":"ContainerStarted","Data":"0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e"}
Oct 30 18:58:12.429139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.428890    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:12.442766 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.436964    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:12.569113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:12.566554    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv" podStartSLOduration=2.566532056 podStartE2EDuration="2.566532056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:12.495281597 +0000 UTC m=+1512.758746546" watchObservedRunningTime="2026-10-30 18:58:12.566532056 +0000 UTC m=+1512.829997006"
Oct 30 18:58:13.535113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:13.521781    5065 generic.go:356] "Generic (PLEG): container finished" podID="28eb5945-f438-4b9f-9f18-4fef919850bf" containerID="2d7cf662dbdb21738ed5e091629bea4e478c286140c18a9d1b7f9760b1fe8860" exitCode=0
Oct 30 18:58:13.538948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:13.537870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv" event={"ID":"28eb5945-f438-4b9f-9f18-4fef919850bf","Type":"ContainerDied","Data":"2d7cf662dbdb21738ed5e091629bea4e478c286140c18a9d1b7f9760b1fe8860"}
Oct 30 18:58:14.120418 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.120384    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:58:14.159116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.134350    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" containerID="cri-o://806d216febacd8d6b5ab288f5ef3dbd5fc325145fcf77b120f7bf2d1caa1fce9" gracePeriod=30
Oct 30 18:58:14.189948 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.189908    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:58:14.205703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.174579    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:58:14.400252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.400147    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:14.522361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.522293    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:58:14.548183 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.525864    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:58:14.604289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.539286    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:14.809670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:14.805424    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.115114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.114208    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.207114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.206206    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.211110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.208187    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:58:15.211110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.209845    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.211110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.210919    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:58:15.430167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.415332    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.430167 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.415862    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:58:15.540115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.539429    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.540115 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.540005    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:58:15.606584 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.606511    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.607768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.607739    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:58:15.644842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.644803    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.645707 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.645684    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:58:15.724751 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.724635    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 30 18:58:15.806716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.806566    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.807026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:15.806995    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:58:16.228104 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.228048    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.303112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.284747    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:58:16.303112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.284825    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:58:16.307556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.307439    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.428900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.428865    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:16.458437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.458410    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:16.601775 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.601680    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.639750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.639707    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.659625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.659575    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.661370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.661339    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\" (UniqueName: \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\") pod \"28eb5945-f438-4b9f-9f18-4fef919850bf\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") "
Oct 30 18:58:16.795466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.795420    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w" pod "28eb5945-f438-4b9f-9f18-4fef919850bf" (UID: "28eb5945-f438-4b9f-9f18-4fef919850bf"). InnerVolumeSpecName "kube-api-access-psh7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.803336    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\") pod \"28eb5945-f438-4b9f-9f18-4fef919850bf\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") "
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.803422    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\") pod \"28eb5945-f438-4b9f-9f18-4fef919850bf\" (UID: \"28eb5945-f438-4b9f-9f18-4fef919850bf\") "
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.806191    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psh7w\" (UniqueName: \"kubernetes.io/projected/28eb5945-f438-4b9f-9f18-4fef919850bf-kube-api-access-psh7w\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.815955    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca" pod "28eb5945-f438-4b9f-9f18-4fef919850bf" (UID: "28eb5945-f438-4b9f-9f18-4fef919850bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.816366    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca" pod "28eb5945-f438-4b9f-9f18-4fef919850bf" (UID: "28eb5945-f438-4b9f-9f18-4fef919850bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:16.827113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.818743    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:58:16.915373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.915180    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:16.915373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:16.915221    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28eb5945-f438-4b9f-9f18-4fef919850bf-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.276695 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.231722    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 18:58:17.332116 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.328539    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:17.624172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.623608    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:17.651353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.651308    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:58:17.714387 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.658473    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:58:17.742123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.659966    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:17.743680 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.743640    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"
Oct 30 18:58:17.745781 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.745753    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv" event={"ID":"28eb5945-f438-4b9f-9f18-4fef919850bf","Type":"ContainerDied","Data":"0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e"}
Oct 30 18:58:17.788284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.788244    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d27c18b6971a307f5fd3515ecb895a2bd68fed7cdd7a5445822a367ab53939e"
Oct 30 18:58:17.817811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.817345    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:17.876295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.874175    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:17.896111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:17.885056    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:58:18.046445 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.046414    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:58:18.046647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.046472    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:58:18.046647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.046524    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:58:18.046647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.046556    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:58:18.046647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.046603    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:58:18.051576 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.051553    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:58:18.731110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.731033    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"]
Oct 30 18:58:18.733156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.732117    5065 cpu_manager.go:414] "containerMap: removing container" podUID="28eb5945-f438-4b9f-9f18-4fef919850bf" containerName="revoke-keys"
Oct 30 18:58:18.733156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.732140    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eb5945-f438-4b9f-9f18-4fef919850bf" containerName="revoke-keys"
Oct 30 18:58:18.733156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.732334    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eb5945-f438-4b9f-9f18-4fef919850bf" containerName="revoke-keys"
Oct 30 18:58:18.749215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.745710    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"]
Oct 30 18:58:18.749215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.745823    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.751234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.751150    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:18.752689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.752659    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:18.812028 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.811993    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.812472 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.812447    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.812616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.812600    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqjx\" (UniqueName: \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.916109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.913796    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.916109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.913861    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqjx\" (UniqueName: \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.916109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.913973    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.916109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.914882    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.931113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.926854    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:18.939779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:18.939752    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqjx\" (UniqueName: \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\") pod \"maas-api-revoke-sub-0af0ab66-l5c9v\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:19.068473 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.068426    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:19.425438 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.425399    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"]
Oct 30 18:58:19.439665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.439638    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.471113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.450010    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"]
Oct 30 18:58:19.471113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.455191    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdns\" (UniqueName: \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.532127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.522697    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.532127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.522815    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.669169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.646129    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdns\" (UniqueName: \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.669169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.646395    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.669169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.646477    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.692174 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.684814    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.752180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.752032    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdns\" (UniqueName: \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.755363 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.755305    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.755575 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:19.755540    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cbdc2b_0841_4ac7_9c83_55284b4fec35.slice/crio-08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729 WatchSource:0}: Error finding container 08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729: Status 404 returned error can't find the container with id 08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729
Oct 30 18:58:19.762190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.759611    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:19.762421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:19.762367    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"]
Oct 30 18:58:20.024166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.022646    5065 generic.go:356] "Generic (PLEG): container finished" podID="79044e03-a475-4595-b8f3-04e6866d9558" containerID="806d216febacd8d6b5ab288f5ef3dbd5fc325145fcf77b120f7bf2d1caa1fce9" exitCode=0
Oct 30 18:58:20.035375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.035336    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v" event={"ID":"19cbdc2b-0841-4ac7-9c83-55284b4fec35","Type":"ContainerStarted","Data":"08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729"}
Oct 30 18:58:20.035504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.035407    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerDied","Data":"806d216febacd8d6b5ab288f5ef3dbd5fc325145fcf77b120f7bf2d1caa1fce9"}
Oct 30 18:58:20.035504 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.035448    5065 scope.go:122] "RemoveContainer" containerID="3f1cf3ab5cdaf3d95f421d0a6e322f4b2235656a08bfdee41300a637226b329a"
Oct 30 18:58:20.068823 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.068804    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163673    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163744    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163847    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163892    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163892    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163922    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.163950    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164004 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.164004    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.164039    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\") pod \"79044e03-a475-4595-b8f3-04e6866d9558\" (UID: \"79044e03-a475-4595-b8f3-04e6866d9558\") "
Oct 30 18:58:20.164718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.164583    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.165298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.165112    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:20.165536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.165472    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:20.165663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.165638    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:20.165898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.165874    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:20.178663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.175172    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"]
Oct 30 18:58:20.178663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.175486    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "kube-api-access-5xz6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:20.178663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.175594    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:20.178968 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.178939    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:20.193666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.179047    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token" pod "79044e03-a475-4595-b8f3-04e6866d9558" (UID: "79044e03-a475-4595-b8f3-04e6866d9558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:20.193666 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:20.179296    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533a1d11_51ba_4d70_864e_11f371ea99e3.slice/crio-baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2 WatchSource:0}: Error finding container baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2: Status 404 returned error can't find the container with id baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2
Oct 30 18:58:20.265563 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265540    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xz6g\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-kube-api-access-5xz6g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265566    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265578    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79044e03-a475-4595-b8f3-04e6866d9558-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265587    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79044e03-a475-4595-b8f3-04e6866d9558-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265596    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79044e03-a475-4595-b8f3-04e6866d9558-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265605    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265615    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:20.265663 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:20.265624    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79044e03-a475-4595-b8f3-04e6866d9558-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:21.035214 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.035181    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"
Oct 30 18:58:21.035694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.035221    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b" event={"ID":"79044e03-a475-4595-b8f3-04e6866d9558","Type":"ContainerDied","Data":"957b3cb4ae2a7c06fcca30f384c83dbc0afa3d9e518d7178c13125c2d2459879"}
Oct 30 18:58:21.035694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.035279    5065 scope.go:122] "RemoveContainer" containerID="806d216febacd8d6b5ab288f5ef3dbd5fc325145fcf77b120f7bf2d1caa1fce9"
Oct 30 18:58:21.037657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.037628    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j" event={"ID":"533a1d11-51ba-4d70-864e-11f371ea99e3","Type":"ContainerStarted","Data":"a7b4c006e723577416525596b4183c8bfb892ba26b88db776cdf9bfe999b7ec8"}
Oct 30 18:58:21.037798 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.037686    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j" event={"ID":"533a1d11-51ba-4d70-864e-11f371ea99e3","Type":"ContainerStarted","Data":"baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2"}
Oct 30 18:58:21.040033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.040005    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v" event={"ID":"19cbdc2b-0841-4ac7-9c83-55284b4fec35","Type":"ContainerStarted","Data":"715cb99f245bb5aa4683c6cf75436bb62812a548a20516be6a883f3e0fa2c50b"}
Oct 30 18:58:21.121943 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.121910    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:58:21.127657 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.127632    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fa7b20-openshift-default-56cf5c98df-48x8b"]
Oct 30 18:58:21.132281 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.132242    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v" podStartSLOduration=3.132229896 podStartE2EDuration="3.132229896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:21.11852736 +0000 UTC m=+1521.381992310" watchObservedRunningTime="2026-10-30 18:58:21.132229896 +0000 UTC m=+1521.395694847"
Oct 30 18:58:21.144652 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.144614    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j" podStartSLOduration=2.144601286 podStartE2EDuration="2.144601286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:21.14175156 +0000 UTC m=+1521.405216508" watchObservedRunningTime="2026-10-30 18:58:21.144601286 +0000 UTC m=+1521.408066234"
Oct 30 18:58:21.895706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:21.895616    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79044e03-a475-4595-b8f3-04e6866d9558" path="/var/lib/kubelet/pods/79044e03-a475-4595-b8f3-04e6866d9558/volumes"
Oct 30 18:58:22.055928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:22.055894    5065 generic.go:356] "Generic (PLEG): container finished" podID="533a1d11-51ba-4d70-864e-11f371ea99e3" containerID="a7b4c006e723577416525596b4183c8bfb892ba26b88db776cdf9bfe999b7ec8" exitCode=0
Oct 30 18:58:22.056453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:22.055976    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j" event={"ID":"533a1d11-51ba-4d70-864e-11f371ea99e3","Type":"ContainerDied","Data":"a7b4c006e723577416525596b4183c8bfb892ba26b88db776cdf9bfe999b7ec8"}
Oct 30 18:58:22.057771 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:22.057744    5065 generic.go:356] "Generic (PLEG): container finished" podID="19cbdc2b-0841-4ac7-9c83-55284b4fec35" containerID="715cb99f245bb5aa4683c6cf75436bb62812a548a20516be6a883f3e0fa2c50b" exitCode=0
Oct 30 18:58:22.057917 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:22.057786    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v" event={"ID":"19cbdc2b-0841-4ac7-9c83-55284b4fec35","Type":"ContainerDied","Data":"715cb99f245bb5aa4683c6cf75436bb62812a548a20516be6a883f3e0fa2c50b"}
Oct 30 18:58:23.079480 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:23.078687    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:58:23.439647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.418152    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:23.439647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.429641    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:23.553955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.553914    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\" (UniqueName: \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\") pod \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") "
Oct 30 18:58:23.554206 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554017    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\") pod \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") "
Oct 30 18:58:23.554301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554272    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\" (UniqueName: \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\") pod \"533a1d11-51ba-4d70-864e-11f371ea99e3\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") "
Oct 30 18:58:23.554356 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554347    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\") pod \"533a1d11-51ba-4d70-864e-11f371ea99e3\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") "
Oct 30 18:58:23.554429 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554398    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\") pod \"533a1d11-51ba-4d70-864e-11f371ea99e3\" (UID: \"533a1d11-51ba-4d70-864e-11f371ea99e3\") "
Oct 30 18:58:23.554492 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554438    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca" pod "19cbdc2b-0841-4ac7-9c83-55284b4fec35" (UID: "19cbdc2b-0841-4ac7-9c83-55284b4fec35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:23.554648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554624    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\") pod \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\" (UID: \"19cbdc2b-0841-4ac7-9c83-55284b4fec35\") "
Oct 30 18:58:23.554818 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.554789    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca" pod "533a1d11-51ba-4d70-864e-11f371ea99e3" (UID: "533a1d11-51ba-4d70-864e-11f371ea99e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:23.555532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.555505    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:23.555633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.555535    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:23.557849 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.557813    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca" pod "533a1d11-51ba-4d70-864e-11f371ea99e3" (UID: "533a1d11-51ba-4d70-864e-11f371ea99e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:23.558873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.558849    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx" pod "19cbdc2b-0841-4ac7-9c83-55284b4fec35" (UID: "19cbdc2b-0841-4ac7-9c83-55284b4fec35"). InnerVolumeSpecName "kube-api-access-thqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:23.559012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.558871    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca" pod "19cbdc2b-0841-4ac7-9c83-55284b4fec35" (UID: "19cbdc2b-0841-4ac7-9c83-55284b4fec35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:23.559012 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.558896    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns" pod "533a1d11-51ba-4d70-864e-11f371ea99e3" (UID: "533a1d11-51ba-4d70-864e-11f371ea99e3"). InnerVolumeSpecName "kube-api-access-vsdns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:23.657042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.656964    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqjx\" (UniqueName: \"kubernetes.io/projected/19cbdc2b-0841-4ac7-9c83-55284b4fec35-kube-api-access-thqjx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:23.657042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.656998    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdns\" (UniqueName: \"kubernetes.io/projected/533a1d11-51ba-4d70-864e-11f371ea99e3-kube-api-access-vsdns\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:23.657042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.657012    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533a1d11-51ba-4d70-864e-11f371ea99e3-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:23.657042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:23.657027    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cbdc2b-0841-4ac7-9c83-55284b4fec35-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.102869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.102820    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"
Oct 30 18:58:24.104000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.102841    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j" event={"ID":"533a1d11-51ba-4d70-864e-11f371ea99e3","Type":"ContainerDied","Data":"baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2"}
Oct 30 18:58:24.104271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.104233    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baddf4d1368907bad76f452e512d3f23e62e2eae4f3ea215ffdf7900e27e39c2"
Oct 30 18:58:24.105838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.105697    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v" event={"ID":"19cbdc2b-0841-4ac7-9c83-55284b4fec35","Type":"ContainerDied","Data":"08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729"}
Oct 30 18:58:24.105838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.105777    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ad7adc517cc76df1fdb492c179ef9601f8dbd2e6b93e4657791ed2408fd729"
Oct 30 18:58:24.106291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.105868    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.904639    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"]
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905734    5065 cpu_manager.go:414] "containerMap: removing container" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905755    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905787    5065 cpu_manager.go:414] "containerMap: removing container" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905795    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905813    5065 cpu_manager.go:414] "containerMap: removing container" podUID="19cbdc2b-0841-4ac7-9c83-55284b4fec35" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905820    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cbdc2b-0841-4ac7-9c83-55284b4fec35" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905861    5065 cpu_manager.go:414] "containerMap: removing container" podUID="533a1d11-51ba-4d70-864e-11f371ea99e3" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.905870    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533a1d11-51ba-4d70-864e-11f371ea99e3" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.906014    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.906027    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="79044e03-a475-4595-b8f3-04e6866d9558" containerName="istio-proxy"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.906036    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cbdc2b-0841-4ac7-9c83-55284b4fec35" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.906051    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="533a1d11-51ba-4d70-864e-11f371ea99e3" containerName="revoke-keys"
Oct 30 18:58:24.926420 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.922859    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:24.931148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.931118    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"]
Oct 30 18:58:24.932884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.932132    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:24.932884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:24.932206    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:25.083357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.083320    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csbd\" (UniqueName: \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.083507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.083465    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.083546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.083520    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.185621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.185530    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csbd\" (UniqueName: \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.186144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.185629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.186144 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.185713    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.187322 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.187297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.189958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.189927    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.207354 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.207325    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csbd\" (UniqueName: \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\") pod \"maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.274709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.274663    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:25.789113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:25.785440    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"]
Oct 30 18:58:26.062978 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.062954    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:26.063154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.063014    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:26.127026 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.126999    5065 generic.go:356] "Generic (PLEG): container finished" podID="bf02edf3-6113-4616-9056-c099add9960e" containerID="cc67f8059c2b10c955bccc72ac15758c221de7774752bf59be9e03c4dfe162cf" exitCode=0
Oct 30 18:58:26.127162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.127075    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerDied","Data":"cc67f8059c2b10c955bccc72ac15758c221de7774752bf59be9e03c4dfe162cf"}
Oct 30 18:58:26.127500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.127479    5065 scope.go:122] "RemoveContainer" containerID="cc67f8059c2b10c955bccc72ac15758c221de7774752bf59be9e03c4dfe162cf"
Oct 30 18:58:26.128660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.128635    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd" event={"ID":"2e046aa6-2443-409e-98c8-60d331b56c9e","Type":"ContainerStarted","Data":"d23357cc0a0abb9e4a84e8e7c6c77c6f66c17d4d3a7dc4d20380423b70122c3c"}
Oct 30 18:58:26.128961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.128934    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:26.129160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.129006    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:26.129160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:26.129038    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:27.184218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:27.182561    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerStarted","Data":"792528e1d5b9215160936f79d9eb17ad7c4e6282ad0dfce0023fc52975390e1c"}
Oct 30 18:58:27.187518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:27.187487    5065 generic.go:356] "Generic (PLEG): container finished" podID="2e046aa6-2443-409e-98c8-60d331b56c9e" containerID="cab3446512f30f490233b32e7bc9bf07c148c1f21f858ac511bfc14ff7b4303e" exitCode=0
Oct 30 18:58:27.187826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:27.187808    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd" event={"ID":"2e046aa6-2443-409e-98c8-60d331b56c9e","Type":"ContainerDied","Data":"cab3446512f30f490233b32e7bc9bf07c148c1f21f858ac511bfc14ff7b4303e"}
Oct 30 18:58:28.122589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.122558    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:58:28.122811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.122747    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:28.122880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.122805    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:28.527438 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.527408    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:28.704112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.698508    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\" (UniqueName: \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\") pod \"2e046aa6-2443-409e-98c8-60d331b56c9e\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") "
Oct 30 18:58:28.704112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.698808    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\") pod \"2e046aa6-2443-409e-98c8-60d331b56c9e\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") "
Oct 30 18:58:28.704112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.698878    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\") pod \"2e046aa6-2443-409e-98c8-60d331b56c9e\" (UID: \"2e046aa6-2443-409e-98c8-60d331b56c9e\") "
Oct 30 18:58:28.706839 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.706810    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca" pod "2e046aa6-2443-409e-98c8-60d331b56c9e" (UID: "2e046aa6-2443-409e-98c8-60d331b56c9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:28.708182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.708062    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd" pod "2e046aa6-2443-409e-98c8-60d331b56c9e" (UID: "2e046aa6-2443-409e-98c8-60d331b56c9e"). InnerVolumeSpecName "kube-api-access-8csbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:28.708623 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.708594    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca" pod "2e046aa6-2443-409e-98c8-60d331b56c9e" (UID: "2e046aa6-2443-409e-98c8-60d331b56c9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:28.808447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.808410    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:28.808447 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.808452    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e046aa6-2443-409e-98c8-60d331b56c9e-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:28.808740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:28.808471    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8csbd\" (UniqueName: \"kubernetes.io/projected/2e046aa6-2443-409e-98c8-60d331b56c9e-kube-api-access-8csbd\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.123290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.123203    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:29.123290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.123271    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:29.336122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.334057    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd" event={"ID":"2e046aa6-2443-409e-98c8-60d331b56c9e","Type":"ContainerDied","Data":"d23357cc0a0abb9e4a84e8e7c6c77c6f66c17d4d3a7dc4d20380423b70122c3c"}
Oct 30 18:58:29.336122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.334149    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23357cc0a0abb9e4a84e8e7c6c77c6f66c17d4d3a7dc4d20380423b70122c3c"
Oct 30 18:58:29.336122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.334238    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"
Oct 30 18:58:29.711111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.707714    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:58:29.728633 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:29.728596    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:58:30.125109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:30.124602    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:30.125109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:30.124677    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:31.185145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:31.178671    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:31.194242 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:31.194202    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:32.122617 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:32.122584    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 18:58:32.122807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:32.122642    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 18:58:33.126799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.126771    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:58:33.127372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.127346    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:58:33.128054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.128035    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:58:33.406506 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.406433    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"]
Oct 30 18:58:33.408042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.408012    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2e046aa6-2443-409e-98c8-60d331b56c9e" containerName="revoke-keys"
Oct 30 18:58:33.408042 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.408041    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e046aa6-2443-409e-98c8-60d331b56c9e" containerName="revoke-keys"
Oct 30 18:58:33.408301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.408288    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e046aa6-2443-409e-98c8-60d331b56c9e" containerName="revoke-keys"
Oct 30 18:58:33.414215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.414191    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.419573 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.419548    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:33.419668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.419616    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:33.422421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.422395    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"]
Oct 30 18:58:33.452738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.452700    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.452873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.452839    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.453120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.453079    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5rs\" (UniqueName: \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.556166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.555546    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5rs\" (UniqueName: \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.556166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.555760    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.556166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.555891    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.557509 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.557477    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.560811 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.560761    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.584534 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.584492    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5rs\" (UniqueName: \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\") pod \"maas-api-revoke-sub-cab88ccf-799kt\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:33.733806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:33.733743    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:34.023253 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:34.023220    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e189df7_1e47_487c_8b2f_934ecb60b235.slice/crio-15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975 WatchSource:0}: Error finding container 15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975: Status 404 returned error can't find the container with id 15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975
Oct 30 18:58:34.027636 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.027608    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"]
Oct 30 18:58:34.427129 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.427049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt" event={"ID":"0e189df7-1e47-487c-8b2f-934ecb60b235","Type":"ContainerStarted","Data":"c1bd156b7a08436988f52e04258578aaa973bf51d9df7b52a5e6db4ae8331777"}
Oct 30 18:58:34.427556 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.427174    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt" event={"ID":"0e189df7-1e47-487c-8b2f-934ecb60b235","Type":"ContainerStarted","Data":"15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975"}
Oct 30 18:58:34.453186 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.453144    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt" podStartSLOduration=1.4531310290000001 podStartE2EDuration="1.453131029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:34.447374902 +0000 UTC m=+1534.710839944" watchObservedRunningTime="2026-10-30 18:58:34.453131029 +0000 UTC m=+1534.716595977"
Oct 30 18:58:34.708758 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.708646    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" podUID="c9153812-81d3-44e7-90b0-4e2c4922c240" containerName="payload-processing" containerID="cri-o://fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4" gracePeriod=25
Oct 30 18:58:34.729813 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:34.729777    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" podUID="e26eb598-5229-438c-ab5e-63a432dafd03" containerName="payload-pre-processing" containerID="cri-o://e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df" gracePeriod=25
Oct 30 18:58:35.245888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.238261    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:58:35.317854 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.317817    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\") pod \"c9153812-81d3-44e7-90b0-4e2c4922c240\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") "
Oct 30 18:58:35.318111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.317986    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\" (UniqueName: \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\") pod \"c9153812-81d3-44e7-90b0-4e2c4922c240\" (UID: \"c9153812-81d3-44e7-90b0-4e2c4922c240\") "
Oct 30 18:58:35.324187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.323392    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume" pod "c9153812-81d3-44e7-90b0-4e2c4922c240" (UID: "c9153812-81d3-44e7-90b0-4e2c4922c240"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:35.324187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.323904    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz" pod "c9153812-81d3-44e7-90b0-4e2c4922c240" (UID: "c9153812-81d3-44e7-90b0-4e2c4922c240"). InnerVolumeSpecName "kube-api-access-8xdhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:35.378678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.378653    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:58:35.422109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.422058    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9153812-81d3-44e7-90b0-4e2c4922c240-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:35.422109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.422109    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xdhz\" (UniqueName: \"kubernetes.io/projected/c9153812-81d3-44e7-90b0-4e2c4922c240-kube-api-access-8xdhz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:35.440383 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.440351    5065 generic.go:356] "Generic (PLEG): container finished" podID="e26eb598-5229-438c-ab5e-63a432dafd03" containerID="e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df" exitCode=0
Oct 30 18:58:35.440814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.440396    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"
Oct 30 18:58:35.440814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.440437    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" event={"ID":"e26eb598-5229-438c-ab5e-63a432dafd03","Type":"ContainerDied","Data":"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"}
Oct 30 18:58:35.440814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.440501    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59" event={"ID":"e26eb598-5229-438c-ab5e-63a432dafd03","Type":"ContainerDied","Data":"26115948ceeced9434005e9bbf2f8a0640e67beb6dc0a0d6b62eba6fe6050c3f"}
Oct 30 18:58:35.440814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.440530    5065 scope.go:122] "RemoveContainer" containerID="e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"
Oct 30 18:58:35.442323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.442304    5065 generic.go:356] "Generic (PLEG): container finished" podID="0e189df7-1e47-487c-8b2f-934ecb60b235" containerID="c1bd156b7a08436988f52e04258578aaa973bf51d9df7b52a5e6db4ae8331777" exitCode=0
Oct 30 18:58:35.442598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.442463    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt" event={"ID":"0e189df7-1e47-487c-8b2f-934ecb60b235","Type":"ContainerDied","Data":"c1bd156b7a08436988f52e04258578aaa973bf51d9df7b52a5e6db4ae8331777"}
Oct 30 18:58:35.444033 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.444010    5065 generic.go:356] "Generic (PLEG): container finished" podID="c9153812-81d3-44e7-90b0-4e2c4922c240" containerID="fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4" exitCode=0
Oct 30 18:58:35.444158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.444041    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"
Oct 30 18:58:35.444158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.444081    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" event={"ID":"c9153812-81d3-44e7-90b0-4e2c4922c240","Type":"ContainerDied","Data":"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"}
Oct 30 18:58:35.444245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.444166    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b" event={"ID":"c9153812-81d3-44e7-90b0-4e2c4922c240","Type":"ContainerDied","Data":"5187e51f4e61075928292b2b3ca2266b1c1b9d99fed827097a7202d134eca58e"}
Oct 30 18:58:35.457885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.457861    5065 scope.go:122] "RemoveContainer" containerID="e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"
Oct 30 18:58:35.458527 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:35.458504    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df\": container with ID starting with e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df not found: ID does not exist" containerID="e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"
Oct 30 18:58:35.458628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.458537    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df"} err="failed to get container status \"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df\": rpc error: code = NotFound desc = could not find container \"e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df\": container with ID starting with e628677c01094c136edab3696bd7c196bbad76a25046ee839f6f65ba47ca58df not found: ID does not exist"
Oct 30 18:58:35.458628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.458564    5065 scope.go:122] "RemoveContainer" containerID="fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"
Oct 30 18:58:35.469687 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.469671    5065 scope.go:122] "RemoveContainer" containerID="fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"
Oct 30 18:58:35.469962 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:35.469942    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4\": container with ID starting with fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4 not found: ID does not exist" containerID="fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"
Oct 30 18:58:35.470058 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.469966    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4"} err="failed to get container status \"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4\": rpc error: code = NotFound desc = could not find container \"fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4\": container with ID starting with fd0e0ed37dd8562b14257210d757c3227b4a5c7b2e5ee57a78ecbabbb2eb7cb4 not found: ID does not exist"
Oct 30 18:58:35.495990 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.495963    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:58:35.500913 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.500890    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb19ff-cb7f5ff66-fkn4b"]
Oct 30 18:58:35.522815 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.522792    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\") pod \"e26eb598-5229-438c-ab5e-63a432dafd03\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") "
Oct 30 18:58:35.522944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.522930    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\" (UniqueName: \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\") pod \"e26eb598-5229-438c-ab5e-63a432dafd03\" (UID: \"e26eb598-5229-438c-ab5e-63a432dafd03\") "
Oct 30 18:58:35.523279 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.523237    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume" pod "e26eb598-5229-438c-ab5e-63a432dafd03" (UID: "e26eb598-5229-438c-ab5e-63a432dafd03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:35.523862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.523840    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26eb598-5229-438c-ab5e-63a432dafd03-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:35.525178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.525155    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7" pod "e26eb598-5229-438c-ab5e-63a432dafd03" (UID: "e26eb598-5229-438c-ab5e-63a432dafd03"). InnerVolumeSpecName "kube-api-access-4rsx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:35.625469 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.625429    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsx7\" (UniqueName: \"kubernetes.io/projected/e26eb598-5229-438c-ab5e-63a432dafd03-kube-api-access-4rsx7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:35.768852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.768824    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:58:35.774368 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.774342    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb19ff-f9c49dbff-mgj59"]
Oct 30 18:58:35.892243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.892215    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9153812-81d3-44e7-90b0-4e2c4922c240" path="/var/lib/kubelet/pods/c9153812-81d3-44e7-90b0-4e2c4922c240/volumes"
Oct 30 18:58:35.892698 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:35.892681    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26eb598-5229-438c-ab5e-63a432dafd03" path="/var/lib/kubelet/pods/e26eb598-5229-438c-ab5e-63a432dafd03/volumes"
Oct 30 18:58:36.634442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.634419    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:36.742928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.742852    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\") pod \"0e189df7-1e47-487c-8b2f-934ecb60b235\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") "
Oct 30 18:58:36.742928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.742915    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\" (UniqueName: \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\") pod \"0e189df7-1e47-487c-8b2f-934ecb60b235\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") "
Oct 30 18:58:36.743204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.743117    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\") pod \"0e189df7-1e47-487c-8b2f-934ecb60b235\" (UID: \"0e189df7-1e47-487c-8b2f-934ecb60b235\") "
Oct 30 18:58:36.743666 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.743633    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca" pod "0e189df7-1e47-487c-8b2f-934ecb60b235" (UID: "0e189df7-1e47-487c-8b2f-934ecb60b235"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:36.743892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.743873    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:36.745814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.745765    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca" pod "0e189df7-1e47-487c-8b2f-934ecb60b235" (UID: "0e189df7-1e47-487c-8b2f-934ecb60b235"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:36.745814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.745796    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs" pod "0e189df7-1e47-487c-8b2f-934ecb60b235" (UID: "0e189df7-1e47-487c-8b2f-934ecb60b235"). InnerVolumeSpecName "kube-api-access-nc5rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:36.845237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.845212    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc5rs\" (UniqueName: \"kubernetes.io/projected/0e189df7-1e47-487c-8b2f-934ecb60b235-kube-api-access-nc5rs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:36.845237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:36.845235    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e189df7-1e47-487c-8b2f-934ecb60b235-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:37.471203 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:37.471170    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"
Oct 30 18:58:37.471374 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:37.471204    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt" event={"ID":"0e189df7-1e47-487c-8b2f-934ecb60b235","Type":"ContainerDied","Data":"15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975"}
Oct 30 18:58:37.471374 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:37.471259    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15c4ccd6c8e76b3e080443de2ecaf4de8472b942d855cd869e6092b57b18c975"
Oct 30 18:58:39.993231 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:39.992767    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.095112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.085929    5065 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:58:40.095112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.085995    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.123119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.119632    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"]
Oct 30 18:58:40.177112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.167289    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fb19ff-2mk8j"]
Oct 30 18:58:40.996334 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.996278    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:58:40.998157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:40.998136    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:58:41.239820 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.239772    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.371160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.336910    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.485518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.447809    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:58:41.564143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.559759    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"]
Oct 30 18:58:41.579670 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.579633    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-493721-2wrpd"]
Oct 30 18:58:41.580000 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.579968    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.683628 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.678366    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.710243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.689326    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:58:41.849110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.737433    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:42.014315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:41.999649    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:42.361106 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.360735    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:58:42.492404 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.456317    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:42.525190 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.525077    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e046aa6-2443-409e-98c8-60d331b56c9e" path="/var/lib/kubelet/pods/2e046aa6-2443-409e-98c8-60d331b56c9e/volumes"
Oct 30 18:58:42.526316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.526295    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533a1d11-51ba-4d70-864e-11f371ea99e3" path="/var/lib/kubelet/pods/533a1d11-51ba-4d70-864e-11f371ea99e3/volumes"
Oct 30 18:58:42.527163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.527146    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:58:42.527397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.527383    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:58:42.581339 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.581291    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:42.591512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.591475    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:58:42.713227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.698102    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 18:58:42.731155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.723241    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:58:42.732862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.712684    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:42.734866 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.733232    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:58:42.970755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:42.970644    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:58:43.050274 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.034686    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:58:43.106841 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.100134    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 18:58:43.274107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.267593    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 18:58:43.364512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.364465    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 18:58:43.393112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.377462    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:58:43.393112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.377533    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:43.405572 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.403996    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:58:43.429344 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.427651    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 18:58:43.466294 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.466266    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:58:43.477774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.477751    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 18:58:43.535902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.535827    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:58:43.554996 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:43.547443    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 18:58:48.122692 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.122655    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"]
Oct 30 18:58:48.123544 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123522    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e26eb598-5229-438c-ab5e-63a432dafd03" containerName="payload-pre-processing"
Oct 30 18:58:48.123653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123545    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26eb598-5229-438c-ab5e-63a432dafd03" containerName="payload-pre-processing"
Oct 30 18:58:48.123653 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123643    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c9153812-81d3-44e7-90b0-4e2c4922c240" containerName="payload-processing"
Oct 30 18:58:48.123776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123653    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9153812-81d3-44e7-90b0-4e2c4922c240" containerName="payload-processing"
Oct 30 18:58:48.123776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123670    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0e189df7-1e47-487c-8b2f-934ecb60b235" containerName="revoke-keys"
Oct 30 18:58:48.123776 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123678    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e189df7-1e47-487c-8b2f-934ecb60b235" containerName="revoke-keys"
Oct 30 18:58:48.123937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123850    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26eb598-5229-438c-ab5e-63a432dafd03" containerName="payload-pre-processing"
Oct 30 18:58:48.123937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123876    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9153812-81d3-44e7-90b0-4e2c4922c240" containerName="payload-processing"
Oct 30 18:58:48.123937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.123888    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e189df7-1e47-487c-8b2f-934ecb60b235" containerName="revoke-keys"
Oct 30 18:58:48.128100 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.128057    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.133980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.133706    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:48.133980 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.133823    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:48.138739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.138718    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"]
Oct 30 18:58:48.210362 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.210314    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqp4\" (UniqueName: \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.210703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.210477    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.210703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.210609    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.312043 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.311995    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.312256 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.312191    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqp4\" (UniqueName: \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.312379 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.312361    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.313142 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.313110    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.315629 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.315573    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.325591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.325566    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqp4\" (UniqueName: \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\") pod \"maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.448212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.448116    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:48.913837 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.913805    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:58:48.914376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.914325    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy" containerID="cri-o://dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931" gracePeriod=30
Oct 30 18:58:48.918272 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:48.918246    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"]
Oct 30 18:58:48.932415 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:48.928969    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad6587e_6431_455c_9d6f_f3cf1742538b.slice/crio-6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9 WatchSource:0}: Error finding container 6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9: Status 404 returned error can't find the container with id 6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9
Oct 30 18:58:49.799615 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:49.799582    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj" event={"ID":"0ad6587e-6431-455c-9d6f-f3cf1742538b","Type":"ContainerStarted","Data":"818d86b9dc67e8d2d2ebbae4199250d105d492eda863281649a08d5ea8752dd7"}
Oct 30 18:58:49.800202 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:49.799635    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj" event={"ID":"0ad6587e-6431-455c-9d6f-f3cf1742538b","Type":"ContainerStarted","Data":"6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9"}
Oct 30 18:58:49.844919 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:49.844867    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj" podStartSLOduration=1.8448544390000001 podStartE2EDuration="1.844854439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:49.83941055 +0000 UTC m=+1550.102875490" watchObservedRunningTime="2026-10-30 18:58:49.844854439 +0000 UTC m=+1550.108319387"
Oct 30 18:58:50.753931 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:50.753896    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:58:50.754200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:50.754164    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" containerID="cri-o://df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59" gracePeriod=30
Oct 30 18:58:50.813892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:50.813859    5065 generic.go:356] "Generic (PLEG): container finished" podID="0ad6587e-6431-455c-9d6f-f3cf1742538b" containerID="818d86b9dc67e8d2d2ebbae4199250d105d492eda863281649a08d5ea8752dd7" exitCode=0
Oct 30 18:58:50.814449 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:50.813911    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj" event={"ID":"0ad6587e-6431-455c-9d6f-f3cf1742538b","Type":"ContainerDied","Data":"818d86b9dc67e8d2d2ebbae4199250d105d492eda863281649a08d5ea8752dd7"}
Oct 30 18:58:52.199361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.199326    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:52.297204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.297165    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\" (UniqueName: \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\") pod \"0ad6587e-6431-455c-9d6f-f3cf1742538b\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") "
Oct 30 18:58:52.297397 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.297248    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\") pod \"0ad6587e-6431-455c-9d6f-f3cf1742538b\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") "
Oct 30 18:58:52.297480 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.297424    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\") pod \"0ad6587e-6431-455c-9d6f-f3cf1742538b\" (UID: \"0ad6587e-6431-455c-9d6f-f3cf1742538b\") "
Oct 30 18:58:52.297767 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.297740    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca" pod "0ad6587e-6431-455c-9d6f-f3cf1742538b" (UID: "0ad6587e-6431-455c-9d6f-f3cf1742538b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:52.298241 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.298211    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:52.299640 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.299612    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca" pod "0ad6587e-6431-455c-9d6f-f3cf1742538b" (UID: "0ad6587e-6431-455c-9d6f-f3cf1742538b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:52.299755 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.299658    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4" pod "0ad6587e-6431-455c-9d6f-f3cf1742538b" (UID: "0ad6587e-6431-455c-9d6f-f3cf1742538b"). InnerVolumeSpecName "kube-api-access-7rqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:52.399806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.399770    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqp4\" (UniqueName: \"kubernetes.io/projected/0ad6587e-6431-455c-9d6f-f3cf1742538b-kube-api-access-7rqp4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:52.399806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.399808    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad6587e-6431-455c-9d6f-f3cf1742538b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:52.855409 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.855382    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"
Oct 30 18:58:52.855599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.855419    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj" event={"ID":"0ad6587e-6431-455c-9d6f-f3cf1742538b","Type":"ContainerDied","Data":"6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9"}
Oct 30 18:58:52.855599 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.855477    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6152b3ad5461ee795634122f1a98551484454341eaa3e863d404a8b9c76f20b9"
Oct 30 18:58:52.857589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.857559    5065 generic.go:356] "Generic (PLEG): container finished" podID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerID="c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704" exitCode=0
Oct 30 18:58:52.857718 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.857635    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerDied","Data":"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"}
Oct 30 18:58:52.858036 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.858022    5065 scope.go:122] "RemoveContainer" containerID="c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"
Oct 30 18:58:52.860122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.860051    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:52.860234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.860162    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:52.860292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:52.860219    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:53.070459 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:53.070394    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:58:53.603160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.600335    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"]
Oct 30 18:58:53.623113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.621484    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0ad6587e-6431-455c-9d6f-f3cf1742538b" containerName="revoke-keys"
Oct 30 18:58:53.623113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.621517    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad6587e-6431-455c-9d6f-f3cf1742538b" containerName="revoke-keys"
Oct 30 18:58:53.623113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.621843    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad6587e-6431-455c-9d6f-f3cf1742538b" containerName="revoke-keys"
Oct 30 18:58:53.634345 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.634321    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"]
Oct 30 18:58:53.634522 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.634431    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.640739 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.640697    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:53.641632 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.641602    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:58:53.733658 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.733621    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.733838 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.733782    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8rt\" (UniqueName: \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.733912 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.733897    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.835425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.835391    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.835605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.835494    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8rt\" (UniqueName: \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.835605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.835556    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.836289 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.836262    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.837932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.837893    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.839862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.839837    5065 patch_prober.go:29] interesting pod/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:58:53.839969 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.839893    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:58:53.852806 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.852779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8rt\" (UniqueName: \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\") pod \"maas-api-revoke-sub-d5cdbb56-c4g9c\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:53.873153 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.873073    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerStarted","Data":"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"}
Oct 30 18:58:53.952940 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:53.952916    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:54.322900 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:58:54.322244    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d583a3_bf7d_4256_ad71_1f8bad5d75f0.slice/crio-7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4 WatchSource:0}: Error finding container 7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4: Status 404 returned error can't find the container with id 7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4
Oct 30 18:58:54.322900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.322505    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"]
Oct 30 18:58:54.445032 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.445010    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:58:54.564057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564029    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564125    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564206    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564283    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564305    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564337    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564375    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564411    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564503    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\") pod \"52b3cfc3-3416-4ac7-b939-22f839ad5639\" (UID: \"52b3cfc3-3416-4ac7-b939-22f839ad5639\") "
Oct 30 18:58:54.564995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.564956    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:54.565180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565040    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:54.565180 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565101    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:54.565327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565273    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:54.565327 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565285    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:54.565444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565415    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.565444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565435    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b3cfc3-3416-4ac7-b939-22f839ad5639-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.565732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565451    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.565732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565466    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.565732 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.565483    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.566523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.566499    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:54.566794 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.566776    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:54.567262 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.567239    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:54.567380 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.567327    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667" pod "52b3cfc3-3416-4ac7-b939-22f839ad5639" (UID: "52b3cfc3-3416-4ac7-b939-22f839ad5639"). InnerVolumeSpecName "kube-api-access-gh667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:54.671006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.669310    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh667\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-kube-api-access-gh667\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.671006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.669352    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.671006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.669368    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.671006 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.669383    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b3cfc3-3416-4ac7-b939-22f839ad5639-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:54.872772 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.872736    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:54.873947 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.873918    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 18:58:54.874112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.873968    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 18:58:54.909808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.909324    5065 generic.go:356] "Generic (PLEG): container finished" podID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerID="dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931" exitCode=0
Oct 30 18:58:54.909808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.909660    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerDied","Data":"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"}
Oct 30 18:58:54.909808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.909707    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd" event={"ID":"52b3cfc3-3416-4ac7-b939-22f839ad5639","Type":"ContainerDied","Data":"7d1408e9bf3a03d1ba4d4bdc3eb2fc2a1bf8be2e5be2611ab04433cb66e2b407"}
Oct 30 18:58:54.909808 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.909738    5065 scope.go:122] "RemoveContainer" containerID="dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"
Oct 30 18:58:54.911068 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.909896    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"
Oct 30 18:58:54.914458 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.914433    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c" event={"ID":"21d583a3-bf7d-4256-ad71-1f8bad5d75f0","Type":"ContainerStarted","Data":"7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4"}
Oct 30 18:58:54.926435 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.926419    5065 scope.go:122] "RemoveContainer" containerID="c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"
Oct 30 18:58:54.937132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.937111    5065 scope.go:122] "RemoveContainer" containerID="dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"
Oct 30 18:58:54.937428 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:54.937404    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931\": container with ID starting with dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931 not found: ID does not exist" containerID="dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"
Oct 30 18:58:54.937502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.937442    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931"} err="failed to get container status \"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931\": rpc error: code = NotFound desc = could not find container \"dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931\": container with ID starting with dd9fd3b4cd5dfa834bf235c626595fda70013fd84b2a59ff21cac123e69b9931 not found: ID does not exist"
Oct 30 18:58:54.937502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.937464    5065 scope.go:122] "RemoveContainer" containerID="c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"
Oct 30 18:58:54.937749 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:54.937730    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2\": container with ID starting with c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2 not found: ID does not exist" containerID="c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"
Oct 30 18:58:54.937805 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.937756    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2"} err="failed to get container status \"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2\": rpc error: code = NotFound desc = could not find container \"c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2\": container with ID starting with c579727a712a3a4c5bdb83afa959c0d51fcd2216bb12ff37c86a9d84100beca2 not found: ID does not exist"
Oct 30 18:58:54.942490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.942468    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:58:54.947367 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:54.947345    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb19ff-openshift-default-7cb89f9cdf-kdpjd"]
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.587863    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589547    5065 cpu_manager.go:414] "containerMap: removing container" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589569    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589610    5065 cpu_manager.go:414] "containerMap: removing container" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589620    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589916    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.594120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.589939    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" containerName="istio-proxy"
Oct 30 18:58:55.622693 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.622666    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.628485 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.628451    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 18:58:55.699122 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.698271    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ab7df0-openshift-default-dockercfg-nkhcf\""
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.710998    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711124    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711168    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711201    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711248    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsq2\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711404    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711478    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711604    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.757641 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.711688    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.814790    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.814849    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.814886    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.814933    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsq2\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815154    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815203    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815290    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815362    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815461    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.816557 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.815989    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.840113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.821777    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.840360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.829639    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.841250 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.829710    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.841464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.829891    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.844228 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.844163    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.848814 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.848783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.882127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.878590    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 18:58:55.882127 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.878655    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 18:58:55.890786 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.890756    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.893247 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.893003    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsq2\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\") pod \"e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") " pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:55.943827 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:55.942403    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:56.026661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.017156    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b3cfc3-3416-4ac7-b939-22f839ad5639" path="/var/lib/kubelet/pods/52b3cfc3-3416-4ac7-b939-22f839ad5639/volumes"
Oct 30 18:58:56.026661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.017683    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c" event={"ID":"21d583a3-bf7d-4256-ad71-1f8bad5d75f0","Type":"ContainerStarted","Data":"af61ff5d397fab2ce055462a9785795c11a16a87d50ed424bf0be2ea50010da4"}
Oct 30 18:58:56.646316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.615155    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 18:58:56.651618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.651579    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:56.651759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.651671    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:56.651759 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.651721    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:58:56.816703 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.816643    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:58:56.879310 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.879274    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 18:58:56.879596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.879568    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922151    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922224    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922320    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922408    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922496    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922591    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922679    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922725    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.924031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.922792    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\") pod \"00d7da02-4058-4f2c-af65-ab0409355cfc\" (UID: \"00d7da02-4058-4f2c-af65-ab0409355cfc\") "
Oct 30 18:58:56.925155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.924775    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:56.939335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.939311    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:56.941662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.940847    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:56.941662 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.941073    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:56.943258 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.943236    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:56.949669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.949155    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:56.951291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.951268    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:56.954179 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.954156    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:56.955514 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:56.955489    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz" pod "00d7da02-4058-4f2c-af65-ab0409355cfc" (UID: "00d7da02-4058-4f2c-af65-ab0409355cfc"). InnerVolumeSpecName "kube-api-access-kmklz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:57.024694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024658    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d7da02-4058-4f2c-af65-ab0409355cfc-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024694 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024693    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024709    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024724    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmklz\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-kube-api-access-kmklz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024738    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024751    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024767    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024780    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.024904 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.024792    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d7da02-4058-4f2c-af65-ab0409355cfc-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.058117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.057266    5065 generic.go:356] "Generic (PLEG): container finished" podID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerID="df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59" exitCode=0
Oct 30 18:58:57.058117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.057375    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerDied","Data":"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"}
Oct 30 18:58:57.058117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.057426    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4" event={"ID":"00d7da02-4058-4f2c-af65-ab0409355cfc","Type":"ContainerDied","Data":"5f16843a98588473e09554591a6281bc131967e40bea91e78357bce3617c5e3b"}
Oct 30 18:58:57.058117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.057450    5065 scope.go:122] "RemoveContainer" containerID="df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"
Oct 30 18:58:57.058117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.057566    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"
Oct 30 18:58:57.062737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.062673    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerStarted","Data":"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"}
Oct 30 18:58:57.062958 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.062907    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerStarted","Data":"0c343063c04b5a22174be9df45336c52c65f961fd4383bbce1924e3bd63fe0e7"}
Oct 30 18:58:57.067011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.066893    5065 generic.go:356] "Generic (PLEG): container finished" podID="21d583a3-bf7d-4256-ad71-1f8bad5d75f0" containerID="af61ff5d397fab2ce055462a9785795c11a16a87d50ed424bf0be2ea50010da4" exitCode=0
Oct 30 18:58:57.067534 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.067283    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c" event={"ID":"21d583a3-bf7d-4256-ad71-1f8bad5d75f0","Type":"ContainerDied","Data":"af61ff5d397fab2ce055462a9785795c11a16a87d50ed424bf0be2ea50010da4"}
Oct 30 18:58:57.077464 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.077392    5065 scope.go:122] "RemoveContainer" containerID="834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"
Oct 30 18:58:57.093130 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.091112    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podStartSLOduration=2.091079352 podStartE2EDuration="2.091079352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:57.089549283 +0000 UTC m=+1557.353014244" watchObservedRunningTime="2026-10-30 18:58:57.091079352 +0000 UTC m=+1557.354544301"
Oct 30 18:58:57.132519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.132492    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:58:57.138669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.138643    5065 scope.go:122] "RemoveContainer" containerID="df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"
Oct 30 18:58:57.139146 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:57.139122    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59\": container with ID starting with df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59 not found: ID does not exist" containerID="df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"
Oct 30 18:58:57.139245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.139157    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59"} err="failed to get container status \"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59\": rpc error: code = NotFound desc = could not find container \"df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59\": container with ID starting with df5aee8e448c5880bd5851780f1c661812a73557dd639302abb628f867717d59 not found: ID does not exist"
Oct 30 18:58:57.139245 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.139184    5065 scope.go:122] "RemoveContainer" containerID="834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"
Oct 30 18:58:57.139407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.139381    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-493721-openshift-default-69c85897fc-sl6f4"]
Oct 30 18:58:57.139602 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:58:57.139578    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424\": container with ID starting with 834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424 not found: ID does not exist" containerID="834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"
Oct 30 18:58:57.139688 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.139607    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424"} err="failed to get container status \"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424\": rpc error: code = NotFound desc = could not find container \"834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424\": container with ID starting with 834569c440378cba8e3d9d701304400ba649b2117f8e42659414f42d77198424 not found: ID does not exist"
Oct 30 18:58:57.432437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.432408    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:57.666627 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.666603    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:57.671987 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.671958    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\") pod \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") "
Oct 30 18:58:57.672166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.672099    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\") pod \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") "
Oct 30 18:58:57.672277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.672232    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\" (UniqueName: \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\") pod \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\" (UID: \"21d583a3-bf7d-4256-ad71-1f8bad5d75f0\") "
Oct 30 18:58:57.673150 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.673123    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca" pod "21d583a3-bf7d-4256-ad71-1f8bad5d75f0" (UID: "21d583a3-bf7d-4256-ad71-1f8bad5d75f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:57.675288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.675260    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca" pod "21d583a3-bf7d-4256-ad71-1f8bad5d75f0" (UID: "21d583a3-bf7d-4256-ad71-1f8bad5d75f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:57.675423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.675258    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt" pod "21d583a3-bf7d-4256-ad71-1f8bad5d75f0" (UID: "21d583a3-bf7d-4256-ad71-1f8bad5d75f0"). InnerVolumeSpecName "kube-api-access-fj8rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:57.773696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.773663    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj8rt\" (UniqueName: \"kubernetes.io/projected/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-kube-api-access-fj8rt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.773696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.773695    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.773905 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.773706    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21d583a3-bf7d-4256-ad71-1f8bad5d75f0-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:57.872956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.872881    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 18:58:57.872956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.872937    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 18:58:57.917330 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.917296    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" path="/var/lib/kubelet/pods/00d7da02-4058-4f2c-af65-ab0409355cfc/volumes"
Oct 30 18:58:57.946907 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.946886    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:57.952711 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:57.952689    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:58.107988 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.107953    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"
Oct 30 18:58:58.108608 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.108577    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c" event={"ID":"21d583a3-bf7d-4256-ad71-1f8bad5d75f0","Type":"ContainerDied","Data":"7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4"}
Oct 30 18:58:58.108763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.108626    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7921cf83d2de472a64b5e4f71f8645f65365682e1d8994cc784fac9afa3b4ab4"
Oct 30 18:58:58.108763 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.108650    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:58.110041 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.110017    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:58:58.884213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.881444    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 30 18:58:58.884213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:58.881511    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 30 18:58:59.924325 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:59.924293    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:58:59.926971 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:58:59.926948    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:59:00.900252 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.900215    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 18:59:00.901176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901159    5065 cpu_manager.go:414] "containerMap: removing container" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.901244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901178    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.901244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901209    5065 cpu_manager.go:414] "containerMap: removing container" podUID="21d583a3-bf7d-4256-ad71-1f8bad5d75f0" containerName="revoke-keys"
Oct 30 18:59:00.901244 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901218    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d583a3-bf7d-4256-ad71-1f8bad5d75f0" containerName="revoke-keys"
Oct 30 18:59:00.901407 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901396    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d583a3-bf7d-4256-ad71-1f8bad5d75f0" containerName="revoke-keys"
Oct 30 18:59:00.901444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901422    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.901444 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.901435    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.907140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.907117    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:00.915995 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.915780    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ab7df0\""
Oct 30 18:59:00.922353 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.922318    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 18:59:00.981304 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.981273    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 18:59:00.982105 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.982069    5065 cpu_manager.go:414] "containerMap: removing container" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.982201 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.982107    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d7da02-4058-4f2c-af65-ab0409355cfc" containerName="istio-proxy"
Oct 30 18:59:00.986133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.986115    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:00.993137 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.992458    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ab7df0-dockercfg-rjsz6\""
Oct 30 18:59:00.997312 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.997207    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:00.997876 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:00.997599    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxw6m\" (UniqueName: \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.000655 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.000260    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 18:59:01.186159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.181942    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.186159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.182103    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.186159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.182178    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzpz\" (UniqueName: \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.186159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.182217    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxw6m\" (UniqueName: \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.186159 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.183738    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.227994 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.227926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxw6m\" (UniqueName: \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\") pod \"payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.236271 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.235967    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:01.297131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.290510    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzpz\" (UniqueName: \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.297131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.290895    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.297131 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.291795    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.313624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.313591    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzpz\" (UniqueName: \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\") pod \"payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.592431 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:01.592342    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893e6079_0241_4e01_a668_47d065517eae.slice/crio-6a09641bf4e5d0a3d141d5a10ea4fe1b7b6697eb08bdc93eba83324f11f6c2bb WatchSource:0}: Error finding container 6a09641bf4e5d0a3d141d5a10ea4fe1b7b6697eb08bdc93eba83324f11f6c2bb: Status 404 returned error can't find the container with id 6a09641bf4e5d0a3d141d5a10ea4fe1b7b6697eb08bdc93eba83324f11f6c2bb
Oct 30 18:59:01.595182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.595156    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 18:59:01.619409 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.619316    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:01.749955 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.749912    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 18:59:01.755963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.755942    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:01.775650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.775625    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-304f22\""
Oct 30 18:59:01.776208 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.776189    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-304f22-dockercfg-c55gm\""
Oct 30 18:59:01.781977 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.781954    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 18:59:01.803646 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.803601    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:01.803898 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.803873    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6dg\" (UniqueName: \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:01.804684 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.804666    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 18:59:01.959925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:01.839843    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.099117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.039660    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:02.099117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.039823    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5svk\" (UniqueName: \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.099117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.039954    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.099117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.060014    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:02.115132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.113913    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6dg\" (UniqueName: \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:02.127884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.127276    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 18:59:02.127884 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.127341    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 18:59:02.128981 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.128957    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6dg\" (UniqueName: \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\") pod \"payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:02.242842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.238427    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5svk\" (UniqueName: \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.242842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.238975    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.242842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.240319    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.257860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.257822    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5svk\" (UniqueName: \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\") pod \"payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.285034 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.284998    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" event={"ID":"bc364195-6c9a-4902-89a5-b2150d1f68b7","Type":"ContainerStarted","Data":"80bad14d34d29b7f12553bb6ab301fb930269d461789de584642d3d89dd0f9fd"}
Oct 30 18:59:02.287459 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.287431    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" event={"ID":"893e6079-0241-4e01-a668-47d065517eae","Type":"ContainerStarted","Data":"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"}
Oct 30 18:59:02.287550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.287482    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" event={"ID":"893e6079-0241-4e01-a668-47d065517eae","Type":"ContainerStarted","Data":"6a09641bf4e5d0a3d141d5a10ea4fe1b7b6697eb08bdc93eba83324f11f6c2bb"}
Oct 30 18:59:02.288594 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.288572    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:02.313301 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.312581    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" podStartSLOduration=2.312564558 podStartE2EDuration="2.312564558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:02.309302471 +0000 UTC m=+1562.572767420" watchObservedRunningTime="2026-10-30 18:59:02.312564558 +0000 UTC m=+1562.576029506"
Oct 30 18:59:02.384740 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.384700    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:02.417550 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.417526    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:02.751081 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:02.751053    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5cbdda_fba6_4c54_b196_121d410d97c8.slice/crio-efd2991dfbf13cda1ca0bf5210fe7e713e3abd4a8f5b809763b19745a9d530f1 WatchSource:0}: Error finding container efd2991dfbf13cda1ca0bf5210fe7e713e3abd4a8f5b809763b19745a9d530f1: Status 404 returned error can't find the container with id efd2991dfbf13cda1ca0bf5210fe7e713e3abd4a8f5b809763b19745a9d530f1
Oct 30 18:59:02.752417 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.752394    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 18:59:02.825765 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:02.825735    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347f1c25_bf10_4847_a907_8ccfad8a668f.slice/crio-cb9a08eb179ceed98931b4946ec53194dd24da42ee891200426a274efbe772a6 WatchSource:0}: Error finding container cb9a08eb179ceed98931b4946ec53194dd24da42ee891200426a274efbe772a6: Status 404 returned error can't find the container with id cb9a08eb179ceed98931b4946ec53194dd24da42ee891200426a274efbe772a6
Oct 30 18:59:02.829882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:02.829858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 18:59:03.301538 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.301506    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" event={"ID":"347f1c25-bf10-4847-a907-8ccfad8a668f","Type":"ContainerStarted","Data":"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"}
Oct 30 18:59:03.302050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.301566    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" event={"ID":"347f1c25-bf10-4847-a907-8ccfad8a668f","Type":"ContainerStarted","Data":"cb9a08eb179ceed98931b4946ec53194dd24da42ee891200426a274efbe772a6"}
Oct 30 18:59:03.302050 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.301595    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:03.303532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.303506    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" event={"ID":"fc5cbdda-fba6-4c54-b196-121d410d97c8","Type":"ContainerStarted","Data":"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"}
Oct 30 18:59:03.303736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.303550    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" event={"ID":"fc5cbdda-fba6-4c54-b196-121d410d97c8","Type":"ContainerStarted","Data":"efd2991dfbf13cda1ca0bf5210fe7e713e3abd4a8f5b809763b19745a9d530f1"}
Oct 30 18:59:03.303736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.303596    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:03.305709 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.305678    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" event={"ID":"bc364195-6c9a-4902-89a5-b2150d1f68b7","Type":"ContainerStarted","Data":"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"}
Oct 30 18:59:03.305850 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.305833    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:03.325547 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.325497    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" podStartSLOduration=2.325484131 podStartE2EDuration="2.325484131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:03.322934675 +0000 UTC m=+1563.586399624" watchObservedRunningTime="2026-10-30 18:59:03.325484131 +0000 UTC m=+1563.588949078"
Oct 30 18:59:03.348802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.348762    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" podStartSLOduration=2.348750235 podStartE2EDuration="2.348750235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:03.345620733 +0000 UTC m=+1563.609085680" watchObservedRunningTime="2026-10-30 18:59:03.348750235 +0000 UTC m=+1563.612215220"
Oct 30 18:59:03.370883 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.370843    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" podStartSLOduration=3.370831023 podStartE2EDuration="3.370831023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:03.370528862 +0000 UTC m=+1563.633993840" watchObservedRunningTime="2026-10-30 18:59:03.370831023 +0000 UTC m=+1563.634295971"
Oct 30 18:59:03.840031 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.840001    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"]
Oct 30 18:59:03.844961 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.844938    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.850523 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.850497    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:59:03.850660 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.850564    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:03.860778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.860753    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"]
Oct 30 18:59:03.862276 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.862249    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrq9\" (UniqueName: \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.862394 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.862366    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.862466 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.862419    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.964060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.964031    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrq9\" (UniqueName: \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.964240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.964124    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.964240 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.964158    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.964769 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.964738    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.966690 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.966660    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:03.982689 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:03.982666    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrq9\" (UniqueName: \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\") pod \"maas-api-revoke-sub-107eded0-rgz9z\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:04.164846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:04.164773    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:04.455199 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:04.455171    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786fa9c7_cd56_4216_a5ad_51b8b3a6fa7f.slice/crio-7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b WatchSource:0}: Error finding container 7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b: Status 404 returned error can't find the container with id 7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b
Oct 30 18:59:04.455877 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:04.455852    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"]
Oct 30 18:59:05.331166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:05.331138    5065 generic.go:356] "Generic (PLEG): container finished" podID="786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" containerID="401765e2db9be581f75779862a0b84179e258ea105496a40635874b0fcec723a" exitCode=0
Oct 30 18:59:05.331319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:05.331208    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z" event={"ID":"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f","Type":"ContainerDied","Data":"401765e2db9be581f75779862a0b84179e258ea105496a40635874b0fcec723a"}
Oct 30 18:59:05.331319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:05.331259    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z" event={"ID":"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f","Type":"ContainerStarted","Data":"7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b"}
Oct 30 18:59:06.645894 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.643869    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"]
Oct 30 18:59:06.650304 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.650236    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fc28ee-p9mgj"]
Oct 30 18:59:06.805238 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.805205    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:06.874401 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.874365    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\") pod \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") "
Oct 30 18:59:06.874605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.874436    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\" (UniqueName: \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\") pod \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") "
Oct 30 18:59:06.874605 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.874470    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\") pod \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\" (UID: \"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f\") "
Oct 30 18:59:06.874973 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.874942    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca" pod "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" (UID: "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:06.875390 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.875270    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.878582 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.878554    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca" pod "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" (UID: "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:06.886164 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.883625    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9" pod "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" (UID: "786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f"). InnerVolumeSpecName "kube-api-access-7lrq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:06.982172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.981536    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.982172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:06.981573    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrq9\" (UniqueName: \"kubernetes.io/projected/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f-kube-api-access-7lrq9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:07.399263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:07.399231    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"
Oct 30 18:59:07.399263 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:07.399248    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z" event={"ID":"786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f","Type":"ContainerDied","Data":"7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b"}
Oct 30 18:59:07.399545 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:07.399305    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c1987f64d92f6198b89498cb8388c24719c0b52f6ed7bcf6337fd9c7f869c6b"
Oct 30 18:59:07.996490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:07.996440    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad6587e-6431-455c-9d6f-f3cf1742538b" path="/var/lib/kubelet/pods/0ad6587e-6431-455c-9d6f-f3cf1742538b/volumes"
Oct 30 18:59:09.511319 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:09.511288    5065 scope.go:122] "RemoveContainer" containerID="d579244d4ad94655c47e7c13d48e6df678a9ab8d493af42d73436e6006fbda8e"
Oct 30 18:59:09.598725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:09.598701    5065 scope.go:122] "RemoveContainer" containerID="325d50b7c62d85e4f66c8fc62b26690393a93ca9e6f417620742f5401c846e3f"
Oct 30 18:59:09.648596 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:09.644747    5065 scope.go:122] "RemoveContainer" containerID="ff2693da27b3edafe31a755b7dc04d9942188cd6825c5c3921555b8ae94caded"
Oct 30 18:59:09.681945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:09.681271    5065 scope.go:122] "RemoveContainer" containerID="1f18fa0b4e9fbdfaca1a3746ca5861d6cca5733c85371cbc122be302c5f045ae"
Oct 30 18:59:14.351156 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.350278    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 18:59:14.366114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.361357    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 18:59:14.366114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.363708    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 18:59:14.366526 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.366505    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 18:59:14.559976 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.559942    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 18:59:14.561060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.561036    5065 cpu_manager.go:414] "containerMap: removing container" podUID="786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" containerName="revoke-keys"
Oct 30 18:59:14.561060 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.561058    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" containerName="revoke-keys"
Oct 30 18:59:14.561277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.561243    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" containerName="revoke-keys"
Oct 30 18:59:14.567349 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.567327    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.601783 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.601719    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 18:59:14.626227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626193    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626265    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626363    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626421    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626461    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626511    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626752 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626724    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626862 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626841    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.626946 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.626927    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.639577 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.639551    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-03362d-openshift-default-dockercfg-rgbtn\""
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.755629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.755724    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.755808    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.755888    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.755946    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.756053    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.756137    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.756180    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.756231    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.756989    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.757507    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.760822    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.764133 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.761058    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.819313 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.809015    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.820602 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.820559    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.854157 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.840004    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.882113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.879438    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.882113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.879907    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:14.890112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:14.888365    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:15.322583 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.322546    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:15.322712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.322623    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:15.322712 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.322662    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:15.324804 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.324780    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 18:59:15.536902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.531867    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:59:15.536902 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.533519    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" containerID="cri-o://792528e1d5b9215160936f79d9eb17ad7c4e6282ad0dfce0023fc52975390e1c" gracePeriod=30
Oct 30 18:59:15.589964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.589876    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerStarted","Data":"ac55f3d29a8d17ea79496afc7557052e0aeddfbe3b58b35d6e4d859d32da18cd"}
Oct 30 18:59:15.589964 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.589923    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerStarted","Data":"1d911cbf9ecf670bb03342717df06092e6dd8e53a979ce8f30ab31ab24211411"}
Oct 30 18:59:15.704112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.697249    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podStartSLOduration=1.697228296 podStartE2EDuration="1.697228296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:15.675447585 +0000 UTC m=+1575.938912551" watchObservedRunningTime="2026-10-30 18:59:15.697228296 +0000 UTC m=+1575.960693244"
Oct 30 18:59:15.931182 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:15.924044    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:16.067580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.067544    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"]
Oct 30 18:59:16.076406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.076379    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.083200 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.083173    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:16.084218 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.084194    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:59:16.110733 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.110697    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"]
Oct 30 18:59:16.157372 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.157338    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg26\" (UniqueName: \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.157528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.157392    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.157661 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.157637    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.259461 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.259430    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.259867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.259565    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg26\" (UniqueName: \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.259867 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.259612    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.260607 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.260579    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.262892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.262872    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.283375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.283344    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg26\" (UniqueName: \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\") pod \"maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.412111 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.401285    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:16.896437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.896406    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:16.959548 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:16.959521    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4070d2_bf24_4175_b591_67800cb3cea9.slice/crio-3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a WatchSource:0}: Error finding container 3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a: Status 404 returned error can't find the container with id 3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a
Oct 30 18:59:16.961064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:16.961040    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"]
Oct 30 18:59:17.672291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:17.672187    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c" event={"ID":"bb4070d2-bf24-4175-b591-67800cb3cea9","Type":"ContainerStarted","Data":"729313b5b180ea09b9331c0dc7a2a72e69521222cefbc32eb61a7666e3a56866"}
Oct 30 18:59:17.672291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:17.672255    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c" event={"ID":"bb4070d2-bf24-4175-b591-67800cb3cea9","Type":"ContainerStarted","Data":"3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a"}
Oct 30 18:59:17.672611 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:17.672588    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:17.673604 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:17.673581    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 18:59:17.696352 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:17.696304    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c" podStartSLOduration=1.696290396 podStartE2EDuration="1.696290396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:17.694457213 +0000 UTC m=+1577.957922174" watchObservedRunningTime="2026-10-30 18:59:17.696290396 +0000 UTC m=+1577.959755379"
Oct 30 18:59:18.129892 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.129855    5065 patch_prober.go:29] interesting pod/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 18:59:18.130396 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.129921    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:59:18.683199 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.683162    5065 generic.go:356] "Generic (PLEG): container finished" podID="bb4070d2-bf24-4175-b591-67800cb3cea9" containerID="729313b5b180ea09b9331c0dc7a2a72e69521222cefbc32eb61a7666e3a56866" exitCode=0
Oct 30 18:59:18.683389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.683250    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c" event={"ID":"bb4070d2-bf24-4175-b591-67800cb3cea9","Type":"ContainerDied","Data":"729313b5b180ea09b9331c0dc7a2a72e69521222cefbc32eb61a7666e3a56866"}
Oct 30 18:59:18.697007 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.696973    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:59:18.697220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.697198    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy" containerID="cri-o://42f3dd9a042ce3436247644a40984886cd7aa239201e6e2d0acc6240be55eb55" gracePeriod=30
Oct 30 18:59:18.783982 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.783950    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:59:18.784213 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:18.784172    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy" containerID="cri-o://890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f" gracePeriod=30
Oct 30 18:59:19.984067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:19.984037    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:20.080512 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.080455    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\") pod \"bb4070d2-bf24-4175-b591-67800cb3cea9\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") "
Oct 30 18:59:20.080683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.080582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\") pod \"bb4070d2-bf24-4175-b591-67800cb3cea9\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") "
Oct 30 18:59:20.080683 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.080675    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\" (UniqueName: \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\") pod \"bb4070d2-bf24-4175-b591-67800cb3cea9\" (UID: \"bb4070d2-bf24-4175-b591-67800cb3cea9\") "
Oct 30 18:59:20.081216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.081192    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca" pod "bb4070d2-bf24-4175-b591-67800cb3cea9" (UID: "bb4070d2-bf24-4175-b591-67800cb3cea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:20.081347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.081316    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:20.083290 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.083259    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca" pod "bb4070d2-bf24-4175-b591-67800cb3cea9" (UID: "bb4070d2-bf24-4175-b591-67800cb3cea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:20.083381 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.083322    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26" pod "bb4070d2-bf24-4175-b591-67800cb3cea9" (UID: "bb4070d2-bf24-4175-b591-67800cb3cea9"). InnerVolumeSpecName "kube-api-access-4kg26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:20.182236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.182208    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg26\" (UniqueName: \"kubernetes.io/projected/bb4070d2-bf24-4175-b591-67800cb3cea9-kube-api-access-4kg26\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:20.182236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.182237    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4070d2-bf24-4175-b591-67800cb3cea9-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:20.763925 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.763889    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c" event={"ID":"bb4070d2-bf24-4175-b591-67800cb3cea9","Type":"ContainerDied","Data":"3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a"}
Oct 30 18:59:20.770737 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.770713    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3776fb7788b7ca751d453df610b3b650dc2abc9de2bc3dd397dd3bf34feb553a"
Oct 30 18:59:20.770963 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.767236    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"
Oct 30 18:59:20.796686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.796651    5065 generic.go:356] "Generic (PLEG): container finished" podID="bf02edf3-6113-4616-9056-c099add9960e" containerID="792528e1d5b9215160936f79d9eb17ad7c4e6282ad0dfce0023fc52975390e1c" exitCode=0
Oct 30 18:59:20.796999 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.796975    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerDied","Data":"792528e1d5b9215160936f79d9eb17ad7c4e6282ad0dfce0023fc52975390e1c"}
Oct 30 18:59:20.797215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:20.797187    5065 scope.go:122] "RemoveContainer" containerID="cc67f8059c2b10c955bccc72ac15758c221de7774752bf59be9e03c4dfe162cf"
Oct 30 18:59:21.067900 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.067871    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:59:21.102731 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102671    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.102882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102736    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.102882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102806    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.102882 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102869    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102960    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103008 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.102990    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103050    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103163 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103128    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103249 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103234    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\") pod \"bf02edf3-6113-4616-9056-c099add9960e\" (UID: \"bf02edf3-6113-4616-9056-c099add9960e\") "
Oct 30 18:59:21.103533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103473    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:21.103648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103628    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:21.103975 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103950    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.104057 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.103982    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.104227 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.104170    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:21.104360 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.104339    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:21.104686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.104663    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:21.105347 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.105326    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "kube-api-access-kxz7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:21.106212 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.106186    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:21.106302 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.106216    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:21.106812 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.106788    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy" pod "bf02edf3-6113-4616-9056-c099add9960e" (UID: "bf02edf3-6113-4616-9056-c099add9960e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:21.205950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205889    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.205950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205951    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf02edf3-6113-4616-9056-c099add9960e-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.205950 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205962    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.206197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205973    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf02edf3-6113-4616-9056-c099add9960e-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.206197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205982    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxz7j\" (UniqueName: \"kubernetes.io/projected/bf02edf3-6113-4616-9056-c099add9960e-kube-api-access-kxz7j\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.206197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.205992    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.206197 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.206001    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf02edf3-6113-4616-9056-c099add9960e-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:21.810612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.810583    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"
Oct 30 18:59:21.810612 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.810603    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b" event={"ID":"bf02edf3-6113-4616-9056-c099add9960e","Type":"ContainerDied","Data":"30379d26c6ded16d20dcc8f1d71492b62216313b947e36667f72d7d2f0413f08"}
Oct 30 18:59:21.810855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.810664    5065 scope.go:122] "RemoveContainer" containerID="792528e1d5b9215160936f79d9eb17ad7c4e6282ad0dfce0023fc52975390e1c"
Oct 30 18:59:21.855391 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.855364    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:59:21.863077 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.863052    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fc28ee-openshift-default-574b7946df-skm5b"]
Oct 30 18:59:21.892856 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.892830    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf02edf3-6113-4616-9056-c099add9960e" path="/var/lib/kubelet/pods/bf02edf3-6113-4616-9056-c099add9960e/volumes"
Oct 30 18:59:21.921705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.921682    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 18:59:21.922497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922467    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.922497 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922487    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.922669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922518    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bb4070d2-bf24-4175-b591-67800cb3cea9" containerName="revoke-keys"
Oct 30 18:59:21.922669 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922525    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4070d2-bf24-4175-b591-67800cb3cea9" containerName="revoke-keys"
Oct 30 18:59:21.922774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922737    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.922774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.922758    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4070d2-bf24-4175-b591-67800cb3cea9" containerName="revoke-keys"
Oct 30 18:59:21.940158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.940130    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:21.942918 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.942894    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 18:59:21.943932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.943910    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.943932 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.943933    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.944158 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.944112    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf02edf3-6113-4616-9056-c099add9960e" containerName="istio-proxy"
Oct 30 18:59:21.950530 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.949641    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:21.952735 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.952716    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c3a5b9-dockercfg-qh24r\""
Oct 30 18:59:21.953067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.953050    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c3a5b9\""
Oct 30 18:59:21.961038 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.961012    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 18:59:21.969316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:21.969290    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 18:59:22.020336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.020307    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwnj\" (UniqueName: \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.020527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.020505    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.020642 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.020607    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.020715 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.020674    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkx7\" (UniqueName: \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.133560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.131156    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkx7\" (UniqueName: \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.133560 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.131367    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwnj\" (UniqueName: \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.154134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.138176    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.154134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.138289    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.154134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.139254    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.154134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.140313    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.200120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.180066    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwnj\" (UniqueName: \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\") pod \"payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.200120 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.186710    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkx7\" (UniqueName: \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\") pod \"payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.295885 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.295853    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.319851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.319824    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.603942 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.603911    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 18:59:22.604579 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:22.604554    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c58cd20_bd59_4bd2_b135_7bfbf5eb008c.slice/crio-7fdc20175754255a45a0d09ebfa95690fec602a6142aec1c86d365228acdccf8 WatchSource:0}: Error finding container 7fdc20175754255a45a0d09ebfa95690fec602a6142aec1c86d365228acdccf8: Status 404 returned error can't find the container with id 7fdc20175754255a45a0d09ebfa95690fec602a6142aec1c86d365228acdccf8
Oct 30 18:59:22.713370 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.713328    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 18:59:22.821774 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.821736    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" event={"ID":"e019ae70-ddae-41c3-94ad-24364cfd4675","Type":"ContainerStarted","Data":"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"}
Oct 30 18:59:22.821960 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.821796    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" event={"ID":"e019ae70-ddae-41c3-94ad-24364cfd4675","Type":"ContainerStarted","Data":"d2e45eb5e126bdddd3604db3b234fc5f39626ae6c0ad044ea8bbd5e006bd164c"}
Oct 30 18:59:22.822134 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.822071    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:22.825722 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.825699    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" event={"ID":"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c","Type":"ContainerStarted","Data":"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"}
Oct 30 18:59:22.825842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.825744    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" event={"ID":"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c","Type":"ContainerStarted","Data":"7fdc20175754255a45a0d09ebfa95690fec602a6142aec1c86d365228acdccf8"}
Oct 30 18:59:22.825842 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.825791    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:22.847486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.847437    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" podStartSLOduration=1.847421025 podStartE2EDuration="1.847421025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:22.842154503 +0000 UTC m=+1583.105619475" watchObservedRunningTime="2026-10-30 18:59:22.847421025 +0000 UTC m=+1583.110885973"
Oct 30 18:59:22.865435 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:22.865397    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" podStartSLOduration=1.865386865 podStartE2EDuration="1.865386865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:22.863565369 +0000 UTC m=+1583.127030317" watchObservedRunningTime="2026-10-30 18:59:22.865386865 +0000 UTC m=+1583.128851812"
Oct 30 18:59:23.071524 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:23.071488    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:59:23.839502 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:23.839473    5065 generic.go:356] "Generic (PLEG): container finished" podID="09939065-c860-48e5-9c62-6b646046638c" containerID="42f3dd9a042ce3436247644a40984886cd7aa239201e6e2d0acc6240be55eb55" exitCode=0
Oct 30 18:59:23.839872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:23.839548    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" event={"ID":"09939065-c860-48e5-9c62-6b646046638c","Type":"ContainerDied","Data":"42f3dd9a042ce3436247644a40984886cd7aa239201e6e2d0acc6240be55eb55"}
Oct 30 18:59:24.039675 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.039647    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:59:24.048716 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.048611    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"]
Oct 30 18:59:24.053768 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.053743    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77071f8-pp657"]
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086151    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086219    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086286    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086399    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086443    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086604    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086667    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086700    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.086781    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\") pod \"09939065-c860-48e5-9c62-6b646046638c\" (UID: \"09939065-c860-48e5-9c62-6b646046638c\") "
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.088070    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.088319    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.088480    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.088955    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:24.091148 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.089137    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.092016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.091413    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:24.098269 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.097574    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.098606 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.098579    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "kube-api-access-d9j84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:24.099588 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.099563    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token" pod "09939065-c860-48e5-9c62-6b646046638c" (UID: "09939065-c860-48e5-9c62-6b646046638c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:24.124756 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.124728    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:59:24.192216 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192185    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192323    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192355    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192400    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192529    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192589    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192650    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192706    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192710    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192773    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\") pod \"0780df06-a621-4798-a156-306814b6ebf4\" (UID: \"0780df06-a621-4798-a156-306814b6ebf4\") "
Oct 30 18:59:24.193132 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.192952    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.193852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.193178    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.193852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.193355    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.193852 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.193376    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:24.194139 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194116    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194144    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09939065-c860-48e5-9c62-6b646046638c-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194160    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194191    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9j84\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-kube-api-access-d9j84\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194217 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194207    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194221    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194235    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194248    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09939065-c860-48e5-9c62-6b646046638c-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194262    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0780df06-a621-4798-a156-306814b6ebf4-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194275    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09939065-c860-48e5-9c62-6b646046638c-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194303    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194317    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09939065-c860-48e5-9c62-6b646046638c-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194330    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.194421 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.194346    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.196166 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.196138    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:24.196997 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.196970    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:24.197123 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.197068    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "kube-api-access-26hhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:24.197315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.197282    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo" pod "0780df06-a621-4798-a156-306814b6ebf4" (UID: "0780df06-a621-4798-a156-306814b6ebf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:24.295901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.295868    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.295901 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.295897    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hhf\" (UniqueName: \"kubernetes.io/projected/0780df06-a621-4798-a156-306814b6ebf4-kube-api-access-26hhf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.296155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.295909    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0780df06-a621-4798-a156-306814b6ebf4-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.296155 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.295918    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0780df06-a621-4798-a156-306814b6ebf4-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:24.850446 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.850412    5065 generic.go:356] "Generic (PLEG): container finished" podID="0780df06-a621-4798-a156-306814b6ebf4" containerID="890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f" exitCode=0
Oct 30 18:59:24.850927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.850464    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"
Oct 30 18:59:24.850927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.850499    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerDied","Data":"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"}
Oct 30 18:59:24.850927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.850555    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn" event={"ID":"0780df06-a621-4798-a156-306814b6ebf4","Type":"ContainerDied","Data":"a278e8125a805da5c5c926a0e6f143afe66e30583ba0e00b80e180e1c070fdd7"}
Oct 30 18:59:24.850927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.850583    5065 scope.go:122] "RemoveContainer" containerID="890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"
Oct 30 18:59:24.852451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.852416    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm" event={"ID":"09939065-c860-48e5-9c62-6b646046638c","Type":"ContainerDied","Data":"50b102a8db703bedaff4d4137a8620225fd73767aca4673cb18ac25a9a180883"}
Oct 30 18:59:24.852451 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.852428    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"
Oct 30 18:59:24.864578 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.864561    5065 scope.go:122] "RemoveContainer" containerID="5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"
Oct 30 18:59:24.875490 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.875470    5065 scope.go:122] "RemoveContainer" containerID="890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"
Oct 30 18:59:24.875772 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:24.875754    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f\": container with ID starting with 890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f not found: ID does not exist" containerID="890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"
Oct 30 18:59:24.875834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.875779    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f"} err="failed to get container status \"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f\": rpc error: code = NotFound desc = could not find container \"890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f\": container with ID starting with 890b48a46e52c5e8fc51e9af57d97ebd95a390a5c9443845fd69206d14585a9f not found: ID does not exist"
Oct 30 18:59:24.875834 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.875796    5065 scope.go:122] "RemoveContainer" containerID="5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"
Oct 30 18:59:24.876032 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:24.876015    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d\": container with ID starting with 5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d not found: ID does not exist" containerID="5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"
Oct 30 18:59:24.876073 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.876036    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d"} err="failed to get container status \"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d\": rpc error: code = NotFound desc = could not find container \"5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d\": container with ID starting with 5aadc9918b8921e91937b33c245989fb001422f9f33ee599cd7e43b98ad7941d not found: ID does not exist"
Oct 30 18:59:24.876073 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.876049    5065 scope.go:122] "RemoveContainer" containerID="42f3dd9a042ce3436247644a40984886cd7aa239201e6e2d0acc6240be55eb55"
Oct 30 18:59:24.896986 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.896962    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:59:24.902471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.902448    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-gpjfn"]
Oct 30 18:59:24.913949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.913928    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:59:24.919070 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:24.919044    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-nwtbm"]
Oct 30 18:59:25.895802 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:25.895762    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0780df06-a621-4798-a156-306814b6ebf4" path="/var/lib/kubelet/pods/0780df06-a621-4798-a156-306814b6ebf4/volumes"
Oct 30 18:59:25.896519 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:25.896494    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09939065-c860-48e5-9c62-6b646046638c" path="/var/lib/kubelet/pods/09939065-c860-48e5-9c62-6b646046638c/volumes"
Oct 30 18:59:25.897145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:25.897123    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c88bab4-0155-4aab-89c3-af0aa8b170c8" path="/var/lib/kubelet/pods/8c88bab4-0155-4aab-89c3-af0aa8b170c8/volumes"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.727141    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728313    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728336    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728379    5065 cpu_manager.go:414] "containerMap: removing container" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728388    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728577    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728594    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.730114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.728604    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="09939065-c860-48e5-9c62-6b646046638c" containerName="istio-proxy"
Oct 30 18:59:26.748110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.737932    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.768109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.765215    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 18:59:26.780796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.780752    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 18:59:26.781598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.781575    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.781598 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.781597    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0780df06-a621-4798-a156-306814b6ebf4" containerName="istio-proxy"
Oct 30 18:59:26.789705 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.788543    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.800916 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.793354    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 18:59:26.872719 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.872687    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.872895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.872740    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.872895 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.872883    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.872913    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.872947    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873015    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873109    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873209 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873158    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873235    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873283    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873329    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873357    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4fnp\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873436 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873421    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873485    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.873616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873502    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7trb\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873568    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873616 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873610    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.873790 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.873685    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.974624 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974592    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974640    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974710    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974736    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974768    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974828    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974865    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974898    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974930    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.974970    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975017    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975062    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4fnp\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975160    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975167    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975191    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975216    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7trb\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975275    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975279    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975325    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975352    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975463    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975529    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.975778 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975556    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.976470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.975872    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.976470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.976013    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.976470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.976212    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.976470 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.976315    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.976639 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.976496    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.977822 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.977783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.977938 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.977927    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.978063 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.978045    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.978533 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.978515    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.991278 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.991218    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4fnp\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.991665 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.991608    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:26.992234 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.992214    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7trb\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:26.994375 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:26.994353    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:27.105631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.105602    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:27.119672 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.119647    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:27.429470 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:27.429308    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5bfa6d_07d8_4c0b_a558_b6322de9eb82.slice/crio-d996d45c15cd6073d981fd8acadcbf888776185460ef6605e8cdc88b539b97b5 WatchSource:0}: Error finding container d996d45c15cd6073d981fd8acadcbf888776185460ef6605e8cdc88b539b97b5: Status 404 returned error can't find the container with id d996d45c15cd6073d981fd8acadcbf888776185460ef6605e8cdc88b539b97b5
Oct 30 18:59:27.431750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.431697    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.431878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.431814    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.431878 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.431860    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.432921 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.432899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 18:59:27.444027 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.444005    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 18:59:27.445856 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:27.445835    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79b9022_78a9_434b_bc78_726c3ba4ce61.slice/crio-626607ede1ad4f3471a434428df79244dbcb9f862d4e77167bb66baef922abe0 WatchSource:0}: Error finding container 626607ede1ad4f3471a434428df79244dbcb9f862d4e77167bb66baef922abe0: Status 404 returned error can't find the container with id 626607ede1ad4f3471a434428df79244dbcb9f862d4e77167bb66baef922abe0
Oct 30 18:59:27.448277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.448252    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.448376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.448310    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.448376 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.448347    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:27.924857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.924810    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" event={"ID":"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82","Type":"ContainerStarted","Data":"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"}
Oct 30 18:59:27.925077 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.924891    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" event={"ID":"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82","Type":"ContainerStarted","Data":"d996d45c15cd6073d981fd8acadcbf888776185460ef6605e8cdc88b539b97b5"}
Oct 30 18:59:27.939873 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.939843    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" event={"ID":"c79b9022-78a9-434b-bc78-726c3ba4ce61","Type":"ContainerStarted","Data":"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"}
Oct 30 18:59:27.940053 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.939897    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" event={"ID":"c79b9022-78a9-434b-bc78-726c3ba4ce61","Type":"ContainerStarted","Data":"626607ede1ad4f3471a434428df79244dbcb9f862d4e77167bb66baef922abe0"}
Oct 30 18:59:27.979055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:27.979008    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podStartSLOduration=1.978995479 podStartE2EDuration="1.978995479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:27.974062766 +0000 UTC m=+1588.237527726" watchObservedRunningTime="2026-10-30 18:59:27.978995479 +0000 UTC m=+1588.242460494"
Oct 30 18:59:28.026064 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.026019    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podStartSLOduration=2.026002488 podStartE2EDuration="2.026002488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:28.021124901 +0000 UTC m=+1588.284589850" watchObservedRunningTime="2026-10-30 18:59:28.026002488 +0000 UTC m=+1588.289467438"
Oct 30 18:59:28.106288 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.106261    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:28.107696 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.107666    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:28.107845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.107732    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:28.120949 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.120913    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:28.122323 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.122300    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:28.122424 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:28.122345    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:29.143110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:29.125494    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:29.143110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:29.125555    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:29.143110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:29.133222    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:29.143110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:29.133277    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:30.109564 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:30.109528    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:30.109858 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:30.109815    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:30.123117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:30.123066    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:30.123438 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:30.123371    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:31.141119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.132795    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:31.141119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.132851    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:31.166110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.154170    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:31.166110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.154228    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:31.331676 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.331639    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 18:59:31.363928 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.363897    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.451518 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.451009    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6d7870-openshift-default-dockercfg-sb8km\""
Oct 30 18:59:31.602108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.575956    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.602432 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.602403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.614118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.602646    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.642187 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.642144    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.642498 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.642469    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.642668 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.642650    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.642869 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.642841    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnxt\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.643055 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.643037    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.643284 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.643265    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.644382 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.586515    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 18:59:31.750750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750656    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750750 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750725    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750791    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750864    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750901    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750940    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.750991 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.750988    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnxt\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.751292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.751037    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.751292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.751129    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.790528 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.787162    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.807766    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.808562    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.808859    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.828112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.825952    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.853176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.831160    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.853176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.832972    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.853176 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.844887    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnxt\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:31.887215 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:31.887164    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:32.108389 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.106541    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:32.278846 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.278787    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:32.283595 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.283551    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:32.283833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.283807    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:32.389453 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.389422    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:32.389625 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.389599    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:32.715178 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:32.683210    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:33.133114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.121499    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:33.146486 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.129596    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:33.146865 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.146817    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:33.154220 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.154176    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:33.154437 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.129214    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:33.485124 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.481050    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:33.705555 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.694752    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 18:59:33.706316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.706289    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:59:33.966551 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:33.966430    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:34.176154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.148731    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:34.176154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.148791    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:34.176154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.149360    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:34.176154 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.149397    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:34.276195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.271531    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:34.276195 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.271889    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:34.434944 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.390191    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:34.542650 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.542521    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:34.569430 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.569295    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:34.578145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.569854    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:34.610112 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.578878    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 18:59:34.673295 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.673268    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 18:59:34.673549 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.673536    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:59:34.673682 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.673672    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 18:59:34.673818 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.673806    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 18:59:34.673933 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.645165    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:34.675143 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.664226    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 18:59:34.687316 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.687280    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"]
Oct 30 18:59:34.845908 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:34.845822    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bcb5af-8sh6c"]
Oct 30 18:59:35.159165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.148432    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:35.159165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.148533    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:35.159165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.149293    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:35.159165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.149345    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:35.300110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.271165    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 18:59:35.447580 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.447243    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:35.455757 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.447591    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:59:35.626929 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:35.626829    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerStarted","Data":"e0645603b888f90fcf89859cf2ba3ea73790d5800f3e7e6f1907b40477fac3db"}
Oct 30 18:59:36.146117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.124919    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:36.146117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.124975    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:36.146117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.125640    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:36.146117 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.125687    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:36.265277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.256241    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4070d2-bf24-4175-b591-67800cb3cea9" path="/var/lib/kubelet/pods/bb4070d2-bf24-4175-b591-67800cb3cea9/volumes"
Oct 30 18:59:36.265277 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.256833    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 18:59:36.669930 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:36.669895    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerStarted","Data":"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"}
Oct 30 18:59:37.114110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.113526    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 18:59:37.114110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.113591    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 18:59:37.128109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.125434    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 18:59:37.128109 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.125491    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 18:59:37.129146 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.128378    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:37.134738 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.134716    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:37.160845 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.160783    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podStartSLOduration=6.160765691 podStartE2EDuration="6.160765691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:36.70461059 +0000 UTC m=+1596.968075569" watchObservedRunningTime="2026-10-30 18:59:37.160765691 +0000 UTC m=+1597.424230640"
Oct 30 18:59:37.722826 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.722796    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:37.737443 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:37.737404    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 18:59:38.798527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:38.798485    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:38.801110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:38.801044    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:38.813361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:38.813334    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 18:59:39.159135 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:39.159011    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:59:39.159416 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:39.159381    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:39.838114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:39.822366    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:39.838114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:39.823831    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:39.838114 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:39.825782    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 18:59:43.023857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:43.023826    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:59:43.024357 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:43.024098    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" containerID="cri-o://d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f" gracePeriod=30
Oct 30 18:59:44.155292 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:44.155255    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:59:44.155706 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:44.155349    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:44.927736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:44.927644    5065 patch_prober.go:29] interesting pod/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:59:44.927736 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:44.927713    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:59:45.023926 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.023886    5065 generic.go:356] "Generic (PLEG): container finished" podID="b80abb5f-43ce-4af7-956a-54165637a734" containerID="06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939" exitCode=0
Oct 30 18:59:45.024169 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.023971    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerDied","Data":"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"}
Oct 30 18:59:45.024527 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.024478    5065 scope.go:122] "RemoveContainer" containerID="06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"
Oct 30 18:59:45.026335 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.026306    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:45.026412 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.026384    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:45.026450 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:45.026439    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:46.037678 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:46.037645    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerStarted","Data":"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"}
Oct 30 18:59:46.948984 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:46.948953    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:59:46.949648 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:46.949623    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 18:59:46.949851 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:46.949831    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 18:59:47.947945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:47.947911    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 18:59:47.948439 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:47.947978    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 18:59:48.468937 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.468874    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:59:48.567315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567278    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567315 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567336    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567402    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567465    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567621 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567579    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567628    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567799 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567759    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567816    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.567915 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.567860    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\") pod \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\" (UID: \"a4cac2b4-a66a-4a5e-b26a-447af4bfb062\") "
Oct 30 18:59:48.568336 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.568302    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:48.568471 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.568452    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.569236 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.569206    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:48.575160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.569347    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:48.575160 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.569432    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:48.575597 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.575575    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:48.578610 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.578585    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:48.589107 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.579285    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "kube-api-access-mmgqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:48.625067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.625032    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:48.625419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.625389    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token" pod "a4cac2b4-a66a-4a5e-b26a-447af4bfb062" (UID: "a4cac2b4-a66a-4a5e-b26a-447af4bfb062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:48.669589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669562    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669589 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669587    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgqm\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-kube-api-access-mmgqm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669597    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669607    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669616    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669624    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669632    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.669728 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.669640    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cac2b4-a66a-4a5e-b26a-447af4bfb062-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.947162 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.947102    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 18:59:48.947346 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.947167    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 18:59:48.977001 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.976613    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 18:59:48.977579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977513    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:48.977579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977528    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:48.977579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977544    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:48.977579 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977550    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:48.977801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977656    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:48.977801 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.977667    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerName="istio-proxy"
Oct 30 18:59:49.019118 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:48.997944    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.052635 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.052604    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 18:59:49.056784 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.056761    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-743e0b-openshift-default-dockercfg-dntsq\""
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119448    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119541    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119572    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119611    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119686    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119818    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119885    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwm8\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119930    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.125113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.119988    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256400    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256538    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256597    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256669    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256698    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256742    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.257172 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.256821    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.368507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.368435    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwm8\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.369011 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.368969    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.389113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.378008    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.389113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.378629    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.389113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.386800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.389113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.387126    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.389113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.388860    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.397226 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.397171    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwm8\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.448521    5065 generic.go:356] "Generic (PLEG): container finished" podID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" containerID="d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f" exitCode=0
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.448760    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerDied","Data":"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"}
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.448809    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt" event={"ID":"a4cac2b4-a66a-4a5e-b26a-447af4bfb062","Type":"ContainerDied","Data":"993ed86f8f94c9eb1d2b87de154e35f9697bceb58512c1f7dce87e67afb7adb9"}
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.448838    5065 scope.go:122] "RemoveContainer" containerID="d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.449025    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"
Oct 30 18:59:49.477872 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.460038    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.507268 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.497120    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.507268 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.498048    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:49.551855 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.551829    5065 scope.go:122] "RemoveContainer" containerID="c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"
Oct 30 18:59:49.564871 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.564846    5065 scope.go:122] "RemoveContainer" containerID="d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"
Oct 30 18:59:49.565187 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:49.565164    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f\": container with ID starting with d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f not found: ID does not exist" containerID="d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"
Oct 30 18:59:49.565275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.565195    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f"} err="failed to get container status \"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f\": rpc error: code = NotFound desc = could not find container \"d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f\": container with ID starting with d7f30207bf8bd3b0210a29586eae7f2f4537026606b44cd62418762f8088d67f not found: ID does not exist"
Oct 30 18:59:49.565275 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.565212    5065 scope.go:122] "RemoveContainer" containerID="c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"
Oct 30 18:59:49.565435 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:49.565417    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704\": container with ID starting with c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704 not found: ID does not exist" containerID="c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"
Oct 30 18:59:49.565500 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.565440    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704"} err="failed to get container status \"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704\": rpc error: code = NotFound desc = could not find container \"c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704\": container with ID starting with c0881f6a74bad0c096413d1ddd7ef6bf98a55fdff1dc54a46ea0b69e760cb704 not found: ID does not exist"
Oct 30 18:59:49.570035 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.569996    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:59:49.576099 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.576058    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bcb5af-openshift-default-56fcc5d55c-h2nbt"]
Oct 30 18:59:49.648425 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.648385    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:50.019168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.977700    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 18:59:50.019168 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:49.977791    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 18:59:50.028760 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:50.028718    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1b7cf8_2b0d_4437_be2d_55631e8c9f13.slice/crio-335e0291d73d18c9e92a3f9fdbfc89218b13a1fb2569dcd7dad06376931d66e2 WatchSource:0}: Error finding container 335e0291d73d18c9e92a3f9fdbfc89218b13a1fb2569dcd7dad06376931d66e2: Status 404 returned error can't find the container with id 335e0291d73d18c9e92a3f9fdbfc89218b13a1fb2569dcd7dad06376931d66e2
Oct 30 18:59:50.030890 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.030866    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cac2b4-a66a-4a5e-b26a-447af4bfb062" path="/var/lib/kubelet/pods/a4cac2b4-a66a-4a5e-b26a-447af4bfb062/volumes"
Oct 30 18:59:50.031779 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.031757    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 18:59:50.032618 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.032583    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:50.032700 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.032664    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:50.032745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.032706    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:50.463591 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.463508    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerStarted","Data":"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"}
Oct 30 18:59:50.463875 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.463853    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerStarted","Data":"335e0291d73d18c9e92a3f9fdbfc89218b13a1fb2569dcd7dad06376931d66e2"}
Oct 30 18:59:50.523384 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.519573    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podStartSLOduration=2.519552898 podStartE2EDuration="2.519552898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:50.49863211 +0000 UTC m=+1610.762097058" watchObservedRunningTime="2026-10-30 18:59:50.519552898 +0000 UTC m=+1610.783017848"
Oct 30 18:59:50.649807 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.648475    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:50.948067 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.948030    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 18:59:50.948298 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:50.948128    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 18:59:51.646457 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.646422    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 18:59:51.655406 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.655375    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.673559 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.667123    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-784fbf-openshift-default-dockercfg-xjmtj\""
Oct 30 18:59:51.675373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.674775    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:51.675373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.675128    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 18:59:51.777105 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777053    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777240    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777291 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777284    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777349    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777423 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777406    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777714 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777636    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.777927 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.777902    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.778080 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.778060    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.778361 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.778340    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmtl4\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.882496    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.882549    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.882773    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.882838    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmtl4\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.882939    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.883061    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.883121    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.883210    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.883268    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.884140    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.884625    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.885452    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.886339    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.887151 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.886574    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.913442 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.913368    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmtl4\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.925076 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.925006    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.926243 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.926192    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:51.947113 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:51.944461    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:52.003860 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.003558    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:52.022141 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.021523    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:59:52.039782 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.038466    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:59:52.054945 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.049785    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 18:59:52.483476 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.483408    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:52.484647 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.484622    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 18:59:52.493704 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:52.493676    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fa508f_c8f1_49d2_a820_bf219fe97fb3.slice/crio-8bcc486c3f3dfbb94efcff7268a95b98ede8c6f84b67a98f1adf17816cc73465 WatchSource:0}: Error finding container 8bcc486c3f3dfbb94efcff7268a95b98ede8c6f84b67a98f1adf17816cc73465: Status 404 returned error can't find the container with id 8bcc486c3f3dfbb94efcff7268a95b98ede8c6f84b67a98f1adf17816cc73465
Oct 30 18:59:52.494796 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.494774    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 18:59:52.496745 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.496707    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:52.496843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.496784    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:52.496843 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:52.496828    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 18:59:53.116133 ip-10-0-131-155 kubenswrapper[5065]: E1030 18:59:53.115767    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 18:59:53.497546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:53.497458    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerStarted","Data":"fdb6b654d4a5d264741fa9e0e275fb190f0b46c14349b012f969e28f53353664"}
Oct 30 18:59:53.497546 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:53.497518    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerStarted","Data":"8bcc486c3f3dfbb94efcff7268a95b98ede8c6f84b67a98f1adf17816cc73465"}
Oct 30 18:59:53.533237 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:53.533181    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podStartSLOduration=2.533166173 podStartE2EDuration="2.533166173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:53.528418243 +0000 UTC m=+1613.791883211" watchObservedRunningTime="2026-10-30 18:59:53.533166173 +0000 UTC m=+1613.796631120"
Oct 30 18:59:54.008725 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.008691    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:54.041833 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.041797    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:54.374331 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.374249    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 18:59:54.379631 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.379539    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.398054 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.398027    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 18:59:54.429119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.422029    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-743e0b-dockercfg-8g6ps\""
Oct 30 18:59:54.429119 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.422116    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-743e0b\""
Oct 30 18:59:54.515204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.496148    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclg4\" (UniqueName: \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.515204 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.496367    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.655161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.627581    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.655161 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.627893    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclg4\" (UniqueName: \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.671400 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.655902    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:54.729165 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.716051    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:54.729532 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.729496    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 18:59:54.738140 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:54.734188    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclg4\" (UniqueName: \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\") pod \"payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") " pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:55.017373 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.017342    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:55.319213 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:55.319183    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708cce75_1b42_4ecc_9de0_abb9853d7a28.slice/crio-185a5658bca8e5ffcd59adc94f0e3533ab4bc1aa8dff87e86aef42163ffe46c5 WatchSource:0}: Error finding container 185a5658bca8e5ffcd59adc94f0e3533ab4bc1aa8dff87e86aef42163ffe46c5: Status 404 returned error can't find the container with id 185a5658bca8e5ffcd59adc94f0e3533ab4bc1aa8dff87e86aef42163ffe46c5
Oct 30 18:59:55.322888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.322858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 18:59:55.724016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.723929    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" event={"ID":"708cce75-1b42-4ecc-9de0-abb9853d7a28","Type":"ContainerStarted","Data":"94271e9e3bfb232d715b085bbbe3de5ed42bdffac7fe0f2d42e288ff7c3e84a1"}
Oct 30 18:59:55.724016 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.723986    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" event={"ID":"708cce75-1b42-4ecc-9de0-abb9853d7a28","Type":"ContainerStarted","Data":"185a5658bca8e5ffcd59adc94f0e3533ab4bc1aa8dff87e86aef42163ffe46c5"}
Oct 30 18:59:55.724465 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.724114    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 18:59:55.756956 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:55.756909    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" podStartSLOduration=1.756895997 podStartE2EDuration="1.756895997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:55.751354711 +0000 UTC m=+1616.014819675" watchObservedRunningTime="2026-10-30 18:59:55.756895997 +0000 UTC m=+1616.020360945"
Oct 30 18:59:57.317857 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.317821    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 18:59:57.324145 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.324125    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.332536 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.332432    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-784fbf\""
Oct 30 18:59:57.332880 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.332857    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-784fbf-dockercfg-jpxl5\""
Oct 30 18:59:57.338051 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.336061    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 18:59:57.390108 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.390056    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxct\" (UniqueName: \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.390419 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.390395    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.507065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.501382    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.507065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.501546    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxct\" (UniqueName: \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.507065 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.502804    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.552110 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.551729    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxct\" (UniqueName: \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\") pod \"payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:57.661507 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:57.661420    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:58.516797 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:58.516746    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 18:59:58.532528 ip-10-0-131-155 kubenswrapper[5065]: W1030 18:59:58.532493    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b06930_7ae3_4289_a0f6_f90b8e9fe6bb.slice/crio-37319da8f3411a3f1df965138381c214c09321cb667dbd7e829a641d551c75fa WatchSource:0}: Error finding container 37319da8f3411a3f1df965138381c214c09321cb667dbd7e829a641d551c75fa: Status 404 returned error can't find the container with id 37319da8f3411a3f1df965138381c214c09321cb667dbd7e829a641d551c75fa
Oct 30 18:59:59.174592 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:59.174549    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" event={"ID":"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb","Type":"ContainerStarted","Data":"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"}
Oct 30 18:59:59.174888 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:59.174861    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" event={"ID":"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb","Type":"ContainerStarted","Data":"37319da8f3411a3f1df965138381c214c09321cb667dbd7e829a641d551c75fa"}
Oct 30 18:59:59.177903 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:59.176268    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 18:59:59.204686 ip-10-0-131-155 kubenswrapper[5065]: I1030 18:59:59.204616    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" podStartSLOduration=2.204591926 podStartE2EDuration="2.204591926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:59.202523781 +0000 UTC m=+1619.465988741" watchObservedRunningTime="2026-10-30 18:59:59.204591926 +0000 UTC m=+1619.468056875"
Oct 30 19:00:00.141399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.141365    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:00.146868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.146840    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.154814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.154782    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:00.159933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.159905    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.164819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.164787    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:00:00.173635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.173599    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:00.194123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.190398    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.217113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.208282    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.217113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.208443    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq7x\" (UniqueName: \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.217113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.208602    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.217113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.208656    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.246514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.246481    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:00.255172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.255139    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.263959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.259585    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314233    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxtd\" (UniqueName: \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314309    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314423    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthbg\" (UniqueName: \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314490    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq7x\" (UniqueName: \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314527    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314611    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314700    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.314746    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.317904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.315759    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.320970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.320939    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.335770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.335732    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:00.336047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.336033    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.342299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.341701    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:00.346079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.346037    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq7x\" (UniqueName: \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\") pod \"maas-api-key-cleanup-29889780-92h7n\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.349788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.349756    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:00.351733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.351705    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.357242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.357156    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:00.368182 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.368143    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:00.375634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.375608    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.419756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.419715    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtd\" (UniqueName: \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.419916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.419850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthbg\" (UniqueName: \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.419980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.419926    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.420037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.420001    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6z6\" (UniqueName: \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.420133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.420047    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.420201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.420160    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.422514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.422452    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.423204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.423181    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.423911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.423888    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:00.443017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.442855    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxtd\" (UniqueName: \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.448187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.448142    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:00.452165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.448360    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.454544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.454510    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthbg\" (UniqueName: \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.457770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.457737    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:00.465318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.465288    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:00.473613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.473578    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:00.489283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.477730    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:00.521911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.514341    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.525969    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.526032    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.526063    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.526349    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fnc\" (UniqueName: \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.526406    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zlj\" (UniqueName: \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.526496    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6z6\" (UniqueName: \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.528228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.527759    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.537994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.537958    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:00.538269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.538111    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.541097 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.541037    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6z6\" (UniqueName: \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.549223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.549197    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:00.611578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.610358    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.628554    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.628660    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.628694    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkgz\" (UniqueName: \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.628731    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.628995    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fnc\" (UniqueName: \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.629261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.629046    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zlj\" (UniqueName: \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.632369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.632335    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.633060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.633005    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.650135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.650043    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fnc\" (UniqueName: \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.659590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.655897    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zlj\" (UniqueName: \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.733577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.733536    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.733791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.733635    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkgz\" (UniqueName: \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.734146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.734125    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:00.734411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.734389    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.754884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.754370    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkgz\" (UniqueName: \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:00.754884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.754659    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:00.837106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.831501    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:00.865509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:00.865248    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:01.195020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.194974    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:01.197434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.197402    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:01.256883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.256838    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerStarted","Data":"304b2ba5b0aa9200ccf423a4c6cf16ef43b919b32a53f54f97b84f7c01904c19"}
Oct 30 19:00:01.259209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.259177    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerStarted","Data":"604d422c1f67b94d3acd52cc8d2adc421237a683dbe6089bd985f54906074298"}
Oct 30 19:00:01.348404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.348372    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:01.357240 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:00:01.355573    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43786fbc_16d8_40b2_8859_798c8897f6b3.slice/crio-7d2f20068c3d68e7b8e8dbbb3dabc04db1740b0816f871f1314317c67cc6beb4 WatchSource:0}: Error finding container 7d2f20068c3d68e7b8e8dbbb3dabc04db1740b0816f871f1314317c67cc6beb4: Status 404 returned error can't find the container with id 7d2f20068c3d68e7b8e8dbbb3dabc04db1740b0816f871f1314317c67cc6beb4
Oct 30 19:00:01.443169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.443064    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:01.447949 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:00:01.446915    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecf4c13_25c3_4f09_a7af_a8b287595fac.slice/crio-2573ccb1a32670f5dfb47020e9916b05ab968a76281a28ee83d5a8736fb45e95 WatchSource:0}: Error finding container 2573ccb1a32670f5dfb47020e9916b05ab968a76281a28ee83d5a8736fb45e95: Status 404 returned error can't find the container with id 2573ccb1a32670f5dfb47020e9916b05ab968a76281a28ee83d5a8736fb45e95
Oct 30 19:00:01.535530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.535438    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:01.621135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.606625    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:01.738482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.738447    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:01.745660 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:00:01.742215    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c92e810_b3f5_4a86_aadb_0ed037558f9b.slice/crio-b79daf48a39acadf6031e62aae67621a01c4f90854bfc11208646efbff459fb0 WatchSource:0}: Error finding container b79daf48a39acadf6031e62aae67621a01c4f90854bfc11208646efbff459fb0: Status 404 returned error can't find the container with id b79daf48a39acadf6031e62aae67621a01c4f90854bfc11208646efbff459fb0
Oct 30 19:00:01.747724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:01.746339    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:02.303476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.302846    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerStarted","Data":"b79daf48a39acadf6031e62aae67621a01c4f90854bfc11208646efbff459fb0"}
Oct 30 19:00:02.321878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.321835    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerStarted","Data":"2573ccb1a32670f5dfb47020e9916b05ab968a76281a28ee83d5a8736fb45e95"}
Oct 30 19:00:02.327213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.326721    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerStarted","Data":"eb6f2d289f2ad07d3c2d60b82664fd9bd067b50df9dfb783d69ef7e8a85fb89f"}
Oct 30 19:00:02.331501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.331465    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerStarted","Data":"35805aad2383de8c5edef714f168de0c1c5ab701c12c6dab48d10a4037308cf7"}
Oct 30 19:00:02.331703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.331517    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerStarted","Data":"7d2f20068c3d68e7b8e8dbbb3dabc04db1740b0816f871f1314317c67cc6beb4"}
Oct 30 19:00:02.334553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.334522    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerStarted","Data":"f2bb0bc7d39ad00f9357a6f68b0a7cfcd9eb25c490c02b6c53b64b6d31889e99"}
Oct 30 19:00:02.336998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.336955    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerStarted","Data":"5a4e90ad6e92757e456393416dff25a6162b5a9d03a80a13af782dc465af8a52"}
Oct 30 19:00:02.340175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.340143    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerStarted","Data":"ab28e449a41657cb801cbf274c710cc3fd813515104680a89ab5468bebff2b19"}
Oct 30 19:00:02.342956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.342909    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerStarted","Data":"88d7b8f041bfafaf67c182c0f49042f69d69d0047ad1d188b71d6364b57d11a9"}
Oct 30 19:00:02.370454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.370381    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" podStartSLOduration=2.370362303 podStartE2EDuration="2.370362303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.361407303 +0000 UTC m=+1622.624872261" watchObservedRunningTime="2026-10-30 19:00:02.370362303 +0000 UTC m=+1622.633827252"
Oct 30 19:00:02.431714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.431639    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" podStartSLOduration=2.4316182299999998 podStartE2EDuration="2.43161823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.39213019 +0000 UTC m=+1622.655595163" watchObservedRunningTime="2026-10-30 19:00:02.43161823 +0000 UTC m=+1622.695083178"
Oct 30 19:00:02.440280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:02.440215    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" podStartSLOduration=2.4401970139999998 podStartE2EDuration="2.440197014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.436185971 +0000 UTC m=+1622.699650931" watchObservedRunningTime="2026-10-30 19:00:02.440197014 +0000 UTC m=+1622.703661963"
Oct 30 19:00:03.358268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.358178    5065 generic.go:356] "Generic (PLEG): container finished" podID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerID="e2a367474791ed05c7b4807446d728a1ef935ea83433a4f2d61a17b0f3f2288d" exitCode=60
Oct 30 19:00:03.359285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.358257    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerDied","Data":"e2a367474791ed05c7b4807446d728a1ef935ea83433a4f2d61a17b0f3f2288d"}
Oct 30 19:00:03.359285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.358578    5065 scope.go:122] "RemoveContainer" containerID="e2a367474791ed05c7b4807446d728a1ef935ea83433a4f2d61a17b0f3f2288d"
Oct 30 19:00:03.369549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.369509    5065 generic.go:356] "Generic (PLEG): container finished" podID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerID="8e7ae60c5f203cc9265199dc9a3beb6e09084d531fd76c3ae8291ed19df2a912" exitCode=60
Oct 30 19:00:03.370440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.370069    5065 scope.go:122] "RemoveContainer" containerID="8e7ae60c5f203cc9265199dc9a3beb6e09084d531fd76c3ae8291ed19df2a912"
Oct 30 19:00:03.370575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.370487    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerDied","Data":"8e7ae60c5f203cc9265199dc9a3beb6e09084d531fd76c3ae8291ed19df2a912"}
Oct 30 19:00:03.372642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.372620    5065 generic.go:356] "Generic (PLEG): container finished" podID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerID="4b7d16d5b80836fc008470ae9af614788580514ed73b44cc5130b98705f83fb3" exitCode=60
Oct 30 19:00:03.372750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.372717    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerDied","Data":"4b7d16d5b80836fc008470ae9af614788580514ed73b44cc5130b98705f83fb3"}
Oct 30 19:00:03.373188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.373172    5065 scope.go:122] "RemoveContainer" containerID="4b7d16d5b80836fc008470ae9af614788580514ed73b44cc5130b98705f83fb3"
Oct 30 19:00:03.375591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.375563    5065 generic.go:356] "Generic (PLEG): container finished" podID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerID="35805aad2383de8c5edef714f168de0c1c5ab701c12c6dab48d10a4037308cf7" exitCode=60
Oct 30 19:00:03.375722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.375704    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerDied","Data":"35805aad2383de8c5edef714f168de0c1c5ab701c12c6dab48d10a4037308cf7"}
Oct 30 19:00:03.376176 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.376160    5065 scope.go:122] "RemoveContainer" containerID="35805aad2383de8c5edef714f168de0c1c5ab701c12c6dab48d10a4037308cf7"
Oct 30 19:00:03.450641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.450605    5065 generic.go:356] "Generic (PLEG): container finished" podID="7a319d16-b93e-473b-8ba6-36e299065035" containerID="9f45a6b9f6c4269acc23711064a2ab3bae63b58269bce764df79f3b3d64d4260" exitCode=60
Oct 30 19:00:03.450852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.450696    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerDied","Data":"9f45a6b9f6c4269acc23711064a2ab3bae63b58269bce764df79f3b3d64d4260"}
Oct 30 19:00:03.451138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.451121    5065 scope.go:122] "RemoveContainer" containerID="9f45a6b9f6c4269acc23711064a2ab3bae63b58269bce764df79f3b3d64d4260"
Oct 30 19:00:03.455317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.455061    5065 generic.go:356] "Generic (PLEG): container finished" podID="a58f3803-937f-4fd3-b716-068f601481a8" containerID="5a4e90ad6e92757e456393416dff25a6162b5a9d03a80a13af782dc465af8a52" exitCode=60
Oct 30 19:00:03.455317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.455162    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerDied","Data":"5a4e90ad6e92757e456393416dff25a6162b5a9d03a80a13af782dc465af8a52"}
Oct 30 19:00:03.455827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.455630    5065 scope.go:122] "RemoveContainer" containerID="5a4e90ad6e92757e456393416dff25a6162b5a9d03a80a13af782dc465af8a52"
Oct 30 19:00:03.458233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.458209    5065 generic.go:356] "Generic (PLEG): container finished" podID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerID="8871e7d19460c1f834ae319a87b59bd5f13147bdd4f390fc655777ade5374218" exitCode=60
Oct 30 19:00:03.458620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.458604    5065 scope.go:122] "RemoveContainer" containerID="8871e7d19460c1f834ae319a87b59bd5f13147bdd4f390fc655777ade5374218"
Oct 30 19:00:03.458968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.458933    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerDied","Data":"8871e7d19460c1f834ae319a87b59bd5f13147bdd4f390fc655777ade5374218"}
Oct 30 19:00:03.461110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.461067    5065 generic.go:356] "Generic (PLEG): container finished" podID="c31e12ab-feee-4579-a928-d98ae54db325" containerID="88d7b8f041bfafaf67c182c0f49042f69d69d0047ad1d188b71d6364b57d11a9" exitCode=60
Oct 30 19:00:03.461268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.461134    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerDied","Data":"88d7b8f041bfafaf67c182c0f49042f69d69d0047ad1d188b71d6364b57d11a9"}
Oct 30 19:00:03.461434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:03.461418    5065 scope.go:122] "RemoveContainer" containerID="88d7b8f041bfafaf67c182c0f49042f69d69d0047ad1d188b71d6364b57d11a9"
Oct 30 19:00:04.476639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.476596    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerStarted","Data":"076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"}
Oct 30 19:00:04.485116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.483192    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerStarted","Data":"8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"}
Oct 30 19:00:04.486962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.486924    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerStarted","Data":"15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"}
Oct 30 19:00:04.492526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.492449    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerStarted","Data":"f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"}
Oct 30 19:00:04.495663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.495620    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerStarted","Data":"b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"}
Oct 30 19:00:04.499871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.499828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerStarted","Data":"065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"}
Oct 30 19:00:04.508702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.507750    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerStarted","Data":"7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"}
Oct 30 19:00:04.510708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.510671    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerStarted","Data":"d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"}
Oct 30 19:00:04.511303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.511248    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" podStartSLOduration=4.511230044 podStartE2EDuration="4.511230044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.501510743 +0000 UTC m=+1624.764975691" watchObservedRunningTime="2026-10-30 19:00:04.511230044 +0000 UTC m=+1624.774694993"
Oct 30 19:00:04.575120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.574993    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" podStartSLOduration=4.574978394 podStartE2EDuration="4.574978394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.569179276 +0000 UTC m=+1624.832644243" watchObservedRunningTime="2026-10-30 19:00:04.574978394 +0000 UTC m=+1624.838443342"
Oct 30 19:00:04.604801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.604734    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" podStartSLOduration=4.6047110060000005 podStartE2EDuration="4.604711006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.6007295 +0000 UTC m=+1624.864194458" watchObservedRunningTime="2026-10-30 19:00:04.604711006 +0000 UTC m=+1624.868175955"
Oct 30 19:00:04.630160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.630041    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" podStartSLOduration=4.6300205420000005 podStartE2EDuration="4.630020542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.62834408 +0000 UTC m=+1624.891809030" watchObservedRunningTime="2026-10-30 19:00:04.630020542 +0000 UTC m=+1624.893485494"
Oct 30 19:00:04.661769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:04.661713    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" podStartSLOduration=4.661695474 podStartE2EDuration="4.661695474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.660251354 +0000 UTC m=+1624.923716304" watchObservedRunningTime="2026-10-30 19:00:04.661695474 +0000 UTC m=+1624.925160421"
Oct 30 19:00:05.528977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.528939    5065 generic.go:356] "Generic (PLEG): container finished" podID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerID="d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78" exitCode=60
Oct 30 19:00:05.529477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.529020    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerDied","Data":"d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"}
Oct 30 19:00:05.529477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.529104    5065 scope.go:122] "RemoveContainer" containerID="e2a367474791ed05c7b4807446d728a1ef935ea83433a4f2d61a17b0f3f2288d"
Oct 30 19:00:05.529477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.529377    5065 scope.go:122] "RemoveContainer" containerID="d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"
Oct 30 19:00:05.530047 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.529786    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g_odh-ai-gateway-infra(0c92e810-b3f5-4a86-aadb-0ed037558f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b"
Oct 30 19:00:05.532268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.532245    5065 generic.go:356] "Generic (PLEG): container finished" podID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerID="076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307" exitCode=60
Oct 30 19:00:05.532368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.532327    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerDied","Data":"076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"}
Oct 30 19:00:05.532563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.532549    5065 scope.go:122] "RemoveContainer" containerID="076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"
Oct 30 19:00:05.532862 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.532836    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6_odh-ai-gateway-infra(6ecf4c13-25c3-4f09-a7af-a8b287595fac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac"
Oct 30 19:00:05.534577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.534559    5065 generic.go:356] "Generic (PLEG): container finished" podID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerID="8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6" exitCode=60
Oct 30 19:00:05.534717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.534668    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerDied","Data":"8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"}
Oct 30 19:00:05.534869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.534853    5065 scope.go:122] "RemoveContainer" containerID="8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"
Oct 30 19:00:05.535231 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.535204    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z_odh-ai-gateway-infra(5ccb560e-b119-41f4-8409-5eb7a8cd5ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6"
Oct 30 19:00:05.537159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.537077    5065 generic.go:356] "Generic (PLEG): container finished" podID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerID="15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711" exitCode=60
Oct 30 19:00:05.537450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.537415    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerDied","Data":"15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"}
Oct 30 19:00:05.537566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.537550    5065 scope.go:122] "RemoveContainer" containerID="15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"
Oct 30 19:00:05.537833 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.537810    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk_odh-ai-gateway-infra(43786fbc-16d8-40b2-8859-798c8897f6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3"
Oct 30 19:00:05.545838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.545813    5065 generic.go:356] "Generic (PLEG): container finished" podID="7a319d16-b93e-473b-8ba6-36e299065035" containerID="f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d" exitCode=60
Oct 30 19:00:05.545938 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.545909    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerDied","Data":"f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"}
Oct 30 19:00:05.546142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.546125    5065 scope.go:122] "RemoveContainer" containerID="f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"
Oct 30 19:00:05.546405 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.546384    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt_odh-ai-gateway-infra(7a319d16-b93e-473b-8ba6-36e299065035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" podUID="7a319d16-b93e-473b-8ba6-36e299065035"
Oct 30 19:00:05.546919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.546902    5065 scope.go:122] "RemoveContainer" containerID="8e7ae60c5f203cc9265199dc9a3beb6e09084d531fd76c3ae8291ed19df2a912"
Oct 30 19:00:05.548801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.548778    5065 generic.go:356] "Generic (PLEG): container finished" podID="a58f3803-937f-4fd3-b716-068f601481a8" containerID="b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702" exitCode=60
Oct 30 19:00:05.548920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.548856    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerDied","Data":"b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"}
Oct 30 19:00:05.549031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.549011    5065 scope.go:122] "RemoveContainer" containerID="b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"
Oct 30 19:00:05.549325 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.549305    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7_odh-ai-gateway-infra(a58f3803-937f-4fd3-b716-068f601481a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" podUID="a58f3803-937f-4fd3-b716-068f601481a8"
Oct 30 19:00:05.551204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.551172    5065 generic.go:356] "Generic (PLEG): container finished" podID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerID="065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407" exitCode=60
Oct 30 19:00:05.551311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.551245    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerDied","Data":"065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"}
Oct 30 19:00:05.551552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.551487    5065 scope.go:122] "RemoveContainer" containerID="065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"
Oct 30 19:00:05.551813 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.551788    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw_odh-ai-gateway-infra(c607ebcf-0806-41dd-90da-2a7b93dd5d3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b"
Oct 30 19:00:05.554063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.554039    5065 generic.go:356] "Generic (PLEG): container finished" podID="c31e12ab-feee-4579-a928-d98ae54db325" containerID="7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05" exitCode=60
Oct 30 19:00:05.554200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.554119    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerDied","Data":"7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"}
Oct 30 19:00:05.554339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.554324    5065 scope.go:122] "RemoveContainer" containerID="7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"
Oct 30 19:00:05.554534 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:05.554517    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-92h7n_odh-ai-gateway-infra(c31e12ab-feee-4579-a928-d98ae54db325)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" podUID="c31e12ab-feee-4579-a928-d98ae54db325"
Oct 30 19:00:05.563845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.563809    5065 scope.go:122] "RemoveContainer" containerID="4b7d16d5b80836fc008470ae9af614788580514ed73b44cc5130b98705f83fb3"
Oct 30 19:00:05.580695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.580662    5065 scope.go:122] "RemoveContainer" containerID="35805aad2383de8c5edef714f168de0c1c5ab701c12c6dab48d10a4037308cf7"
Oct 30 19:00:05.602801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.602777    5065 scope.go:122] "RemoveContainer" containerID="9f45a6b9f6c4269acc23711064a2ab3bae63b58269bce764df79f3b3d64d4260"
Oct 30 19:00:05.626710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.626670    5065 scope.go:122] "RemoveContainer" containerID="5a4e90ad6e92757e456393416dff25a6162b5a9d03a80a13af782dc465af8a52"
Oct 30 19:00:05.643379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.643352    5065 scope.go:122] "RemoveContainer" containerID="8871e7d19460c1f834ae319a87b59bd5f13147bdd4f390fc655777ade5374218"
Oct 30 19:00:05.661642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:05.661612    5065 scope.go:122] "RemoveContainer" containerID="88d7b8f041bfafaf67c182c0f49042f69d69d0047ad1d188b71d6364b57d11a9"
Oct 30 19:00:06.706594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.706529    5065 scope.go:122] "RemoveContainer" containerID="076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"
Oct 30 19:00:06.707231 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.706956    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6_odh-ai-gateway-infra(6ecf4c13-25c3-4f09-a7af-a8b287595fac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac"
Oct 30 19:00:06.816348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.815196    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 19:00:06.817895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.817864    5065 scope.go:122] "RemoveContainer" containerID="8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"
Oct 30 19:00:06.818294 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.818265    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z_odh-ai-gateway-infra(5ccb560e-b119-41f4-8409-5eb7a8cd5ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6"
Oct 30 19:00:06.835143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.824561    5065 scope.go:122] "RemoveContainer" containerID="15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"
Oct 30 19:00:06.835143 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.824797    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk_odh-ai-gateway-infra(43786fbc-16d8-40b2-8859-798c8897f6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3"
Oct 30 19:00:06.835143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.827489    5065 scope.go:122] "RemoveContainer" containerID="f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"
Oct 30 19:00:06.835143 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.827708    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt_odh-ai-gateway-infra(7a319d16-b93e-473b-8ba6-36e299065035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" podUID="7a319d16-b93e-473b-8ba6-36e299065035"
Oct 30 19:00:06.895113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.875990    5065 scope.go:122] "RemoveContainer" containerID="b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"
Oct 30 19:00:06.910264 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.895764    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7_odh-ai-gateway-infra(a58f3803-937f-4fd3-b716-068f601481a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" podUID="a58f3803-937f-4fd3-b716-068f601481a8"
Oct 30 19:00:06.971744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.971653    5065 scope.go:122] "RemoveContainer" containerID="7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"
Oct 30 19:00:06.972234 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.972143    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-92h7n_odh-ai-gateway-infra(c31e12ab-feee-4579-a928-d98ae54db325)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" podUID="c31e12ab-feee-4579-a928-d98ae54db325"
Oct 30 19:00:06.991457 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:06.991325    5065 scope.go:122] "RemoveContainer" containerID="065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"
Oct 30 19:00:06.991841 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:06.991802    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw_odh-ai-gateway-infra(c607ebcf-0806-41dd-90da-2a7b93dd5d3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b"
Oct 30 19:00:07.012129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:07.012053    5065 scope.go:122] "RemoveContainer" containerID="d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"
Oct 30 19:00:07.012771 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:07.012739    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g_odh-ai-gateway-infra(0c92e810-b3f5-4a86-aadb-0ed037558f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b"
Oct 30 19:00:10.060206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.060170    5065 generic.go:356] "Generic (PLEG): container finished" podID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerID="ac55f3d29a8d17ea79496afc7557052e0aeddfbe3b58b35d6e4d859d32da18cd" exitCode=0
Oct 30 19:00:10.060742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.060243    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerDied","Data":"ac55f3d29a8d17ea79496afc7557052e0aeddfbe3b58b35d6e4d859d32da18cd"}
Oct 30 19:00:10.060742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.060618    5065 scope.go:122] "RemoveContainer" containerID="ac55f3d29a8d17ea79496afc7557052e0aeddfbe3b58b35d6e4d859d32da18cd"
Oct 30 19:00:10.062517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.062480    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:10.062646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.062597    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:10.062720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.062645    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:10.201923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.201894    5065 scope.go:122] "RemoveContainer" containerID="b1413ac2739bd0525425594f02c873956c3ff19b92c21c0a4ae8c5bf6c9934e3"
Oct 30 19:00:10.235147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.235076    5065 scope.go:122] "RemoveContainer" containerID="57cd131bb168ec2f1a567902cc05a791d1da1101039e316088e7806207c58321"
Oct 30 19:00:10.259901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:10.259880    5065 scope.go:122] "RemoveContainer" containerID="082c89dc127f91427de82814dd0210be02c6e01f67681d3e9dad4032af0dc986"
Oct 30 19:00:11.072309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:11.072273    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerStarted","Data":"f82fb8963a5bc6bdcaf6863460b5b9625c032803fa693c9154bf3efcac849ef6"}
Oct 30 19:00:11.263467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:11.263443    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 19:00:11.892403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:11.891432    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.892403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:11.891487    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:00:11.918687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:11.918661    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:00:12.897109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:12.892350    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.897109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:12.892421    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:00:13.910116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:13.909571    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.910116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:13.909633    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:00:14.899113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:14.889892    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.899113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:14.889953    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:00:15.885650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:15.885619    5065 scope.go:122] "RemoveContainer" containerID="076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"
Oct 30 19:00:15.897754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:15.890556    5065 scope.go:122] "RemoveContainer" containerID="d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"
Oct 30 19:00:15.899901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:15.899805    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.899901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:15.899851    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:00:16.897173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:16.897048    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:00:16.897952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:16.897820    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:00:16.898801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:16.898778    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:00:17.179064 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.178979    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerStarted","Data":"450d98966b9e3de2784deb2482351d7a281f009eea6fef8d039d76a15307c709"}
Oct 30 19:00:17.182000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.181974    5065 generic.go:356] "Generic (PLEG): container finished" podID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerID="2aa5647079c8862a1ffba9aa898fa00b0a31a3388da11295a0784d8d83f105a1" exitCode=60
Oct 30 19:00:17.182207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.182051    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerDied","Data":"2aa5647079c8862a1ffba9aa898fa00b0a31a3388da11295a0784d8d83f105a1"}
Oct 30 19:00:17.182207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.182171    5065 scope.go:122] "RemoveContainer" containerID="076cdb8c870d180eb014b8559de5555ccdd6ec17b5165787c7b969d025ea5307"
Oct 30 19:00:17.182840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.182823    5065 scope.go:122] "RemoveContainer" containerID="2aa5647079c8862a1ffba9aa898fa00b0a31a3388da11295a0784d8d83f105a1"
Oct 30 19:00:17.183135 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:17.183109    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6_odh-ai-gateway-infra(6ecf4c13-25c3-4f09-a7af-a8b287595fac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac"
Oct 30 19:00:17.882099 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.882051    5065 scope.go:122] "RemoveContainer" containerID="7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"
Oct 30 19:00:17.882307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.882165    5065 scope.go:122] "RemoveContainer" containerID="065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"
Oct 30 19:00:17.882307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:17.882236    5065 scope.go:122] "RemoveContainer" containerID="15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"
Oct 30 19:00:18.207212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.207177    5065 generic.go:356] "Generic (PLEG): container finished" podID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerID="450d98966b9e3de2784deb2482351d7a281f009eea6fef8d039d76a15307c709" exitCode=60
Oct 30 19:00:18.207643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.207366    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerDied","Data":"450d98966b9e3de2784deb2482351d7a281f009eea6fef8d039d76a15307c709"}
Oct 30 19:00:18.207845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.207824    5065 scope.go:122] "RemoveContainer" containerID="d2b6b81e38afaf3e33c3e3982db99e138ed166a705ad9eb5164bbde381affb78"
Oct 30 19:00:18.208326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.208303    5065 scope.go:122] "RemoveContainer" containerID="450d98966b9e3de2784deb2482351d7a281f009eea6fef8d039d76a15307c709"
Oct 30 19:00:18.208745 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:18.208721    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g_odh-ai-gateway-infra(0c92e810-b3f5-4a86-aadb-0ed037558f9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b"
Oct 30 19:00:18.322203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.321930    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:18.343750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.343721    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:18.571193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.571173    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:18.639495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.639463    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\" (UniqueName: \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\") pod \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") "
Oct 30 19:00:18.639659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.639582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\") pod \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\" (UID: \"6ecf4c13-25c3-4f09-a7af-a8b287595fac\") "
Oct 30 19:00:18.639954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.639930    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca" pod "6ecf4c13-25c3-4f09-a7af-a8b287595fac" (UID: "6ecf4c13-25c3-4f09-a7af-a8b287595fac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:18.640262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.640244    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf4c13-25c3-4f09-a7af-a8b287595fac-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.642357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.642320    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg" pod "6ecf4c13-25c3-4f09-a7af-a8b287595fac" (UID: "6ecf4c13-25c3-4f09-a7af-a8b287595fac"). InnerVolumeSpecName "kube-api-access-pthbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.741499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.741474    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthbg\" (UniqueName: \"kubernetes.io/projected/6ecf4c13-25c3-4f09-a7af-a8b287595fac-kube-api-access-pthbg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.882286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.882254    5065 scope.go:122] "RemoveContainer" containerID="8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"
Oct 30 19:00:18.883000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:18.882978    5065 scope.go:122] "RemoveContainer" containerID="b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"
Oct 30 19:00:19.225954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.225918    5065 generic.go:356] "Generic (PLEG): container finished" podID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerID="1ae43f651cc845c16dc53efd9ee1b39ba236efa646c2c722eae761a0a4b3572c" exitCode=60
Oct 30 19:00:19.226447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.226015    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerDied","Data":"1ae43f651cc845c16dc53efd9ee1b39ba236efa646c2c722eae761a0a4b3572c"}
Oct 30 19:00:19.226447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.226130    5065 scope.go:122] "RemoveContainer" containerID="065d002ecf4005f4ed91d8e0ede65ce0ffe70c3c452731317657bfba390ac407"
Oct 30 19:00:19.226639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.226618    5065 scope.go:122] "RemoveContainer" containerID="1ae43f651cc845c16dc53efd9ee1b39ba236efa646c2c722eae761a0a4b3572c"
Oct 30 19:00:19.227069 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:19.227046    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw_odh-ai-gateway-infra(c607ebcf-0806-41dd-90da-2a7b93dd5d3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b"
Oct 30 19:00:19.229128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.229063    5065 generic.go:356] "Generic (PLEG): container finished" podID="c31e12ab-feee-4579-a928-d98ae54db325" containerID="f531f8e2f185e2091d482b7f0f5ef3d878cf990fbcc56a265aa1c40dd2fc9ee9" exitCode=60
Oct 30 19:00:19.229237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.229122    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerDied","Data":"f531f8e2f185e2091d482b7f0f5ef3d878cf990fbcc56a265aa1c40dd2fc9ee9"}
Oct 30 19:00:19.229560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.229540    5065 scope.go:122] "RemoveContainer" containerID="f531f8e2f185e2091d482b7f0f5ef3d878cf990fbcc56a265aa1c40dd2fc9ee9"
Oct 30 19:00:19.229942 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:19.229921    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-92h7n_odh-ai-gateway-infra(c31e12ab-feee-4579-a928-d98ae54db325)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" podUID="c31e12ab-feee-4579-a928-d98ae54db325"
Oct 30 19:00:19.233162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.233141    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6" event={"ID":"6ecf4c13-25c3-4f09-a7af-a8b287595fac","Type":"ContainerDied","Data":"2573ccb1a32670f5dfb47020e9916b05ab968a76281a28ee83d5a8736fb45e95"}
Oct 30 19:00:19.233296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.233202    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"
Oct 30 19:00:19.239735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.239708    5065 generic.go:356] "Generic (PLEG): container finished" podID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerID="64565050724d555a526b9fd98951a3752aa96772f98adfc310fb024e0ab53e80" exitCode=60
Oct 30 19:00:19.239860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.239845    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerDied","Data":"64565050724d555a526b9fd98951a3752aa96772f98adfc310fb024e0ab53e80"}
Oct 30 19:00:19.240305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.240286    5065 scope.go:122] "RemoveContainer" containerID="64565050724d555a526b9fd98951a3752aa96772f98adfc310fb024e0ab53e80"
Oct 30 19:00:19.240643 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:19.240617    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk_odh-ai-gateway-infra(43786fbc-16d8-40b2-8859-798c8897f6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3"
Oct 30 19:00:19.243727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.243691    5065 scope.go:122] "RemoveContainer" containerID="7f063be671c2fc8b9ff9c91b953927cd335be9bc67956b0bbf0682f4270e1b05"
Oct 30 19:00:19.274337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.266401    5065 scope.go:122] "RemoveContainer" containerID="2aa5647079c8862a1ffba9aa898fa00b0a31a3388da11295a0784d8d83f105a1"
Oct 30 19:00:19.292722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.292695    5065 scope.go:122] "RemoveContainer" containerID="15c1453c403afba7a2dc76a0ebb53f4e44f7228a8a7b1c4f972fb53246589711"
Oct 30 19:00:19.298075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.298049    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:19.304705 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.304351    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889780-dddd6"]
Oct 30 19:00:19.509126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.509099    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:19.559221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.559193    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\") pod \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") "
Oct 30 19:00:19.559414 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.559396    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\" (UniqueName: \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\") pod \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\" (UID: \"0c92e810-b3f5-4a86-aadb-0ed037558f9b\") "
Oct 30 19:00:19.559597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.559579    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca" pod "0c92e810-b3f5-4a86-aadb-0ed037558f9b" (UID: "0c92e810-b3f5-4a86-aadb-0ed037558f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:19.560043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.560020    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92e810-b3f5-4a86-aadb-0ed037558f9b-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.561777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.561749    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz" pod "0c92e810-b3f5-4a86-aadb-0ed037558f9b" (UID: "0c92e810-b3f5-4a86-aadb-0ed037558f9b"). InnerVolumeSpecName "kube-api-access-2dkgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:19.661946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.661876    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dkgz\" (UniqueName: \"kubernetes.io/projected/0c92e810-b3f5-4a86-aadb-0ed037558f9b-kube-api-access-2dkgz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.889424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.889398    5065 scope.go:122] "RemoveContainer" containerID="f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"
Oct 30 19:00:19.900355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:19.900325    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" path="/var/lib/kubelet/pods/6ecf4c13-25c3-4f09-a7af-a8b287595fac/volumes"
Oct 30 19:00:20.255747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.255716    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"
Oct 30 19:00:20.256206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.255747    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g" event={"ID":"0c92e810-b3f5-4a86-aadb-0ed037558f9b","Type":"ContainerDied","Data":"b79daf48a39acadf6031e62aae67621a01c4f90854bfc11208646efbff459fb0"}
Oct 30 19:00:20.256206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.255809    5065 scope.go:122] "RemoveContainer" containerID="450d98966b9e3de2784deb2482351d7a281f009eea6fef8d039d76a15307c709"
Oct 30 19:00:20.259440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.259415    5065 generic.go:356] "Generic (PLEG): container finished" podID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerID="feab690d3c8c0aa377aa63fe0814b31b9adc282e0316d9cf60b6ad5721793335" exitCode=60
Oct 30 19:00:20.259571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.259487    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerDied","Data":"feab690d3c8c0aa377aa63fe0814b31b9adc282e0316d9cf60b6ad5721793335"}
Oct 30 19:00:20.260824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.260795    5065 scope.go:122] "RemoveContainer" containerID="feab690d3c8c0aa377aa63fe0814b31b9adc282e0316d9cf60b6ad5721793335"
Oct 30 19:00:20.261250 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:20.261222    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z_odh-ai-gateway-infra(5ccb560e-b119-41f4-8409-5eb7a8cd5ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6"
Oct 30 19:00:20.263984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.263964    5065 generic.go:356] "Generic (PLEG): container finished" podID="a58f3803-937f-4fd3-b716-068f601481a8" containerID="4a5347964f6c9efc023b714767ae57a727862a575f7797764f8a1260fbb0d63b" exitCode=60
Oct 30 19:00:20.264075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.263999    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerDied","Data":"4a5347964f6c9efc023b714767ae57a727862a575f7797764f8a1260fbb0d63b"}
Oct 30 19:00:20.264344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.264331    5065 scope.go:122] "RemoveContainer" containerID="4a5347964f6c9efc023b714767ae57a727862a575f7797764f8a1260fbb0d63b"
Oct 30 19:00:20.264592 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:20.264575    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7_odh-ai-gateway-infra(a58f3803-937f-4fd3-b716-068f601481a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" podUID="a58f3803-937f-4fd3-b716-068f601481a8"
Oct 30 19:00:20.269523 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.269504    5065 scope.go:122] "RemoveContainer" containerID="8d440c2ade5af36797f06330a7733c5108fcbe0876e9017c71f145743e4c3db6"
Oct 30 19:00:20.277970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.277948    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:20.287056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.287035    5065 scope.go:122] "RemoveContainer" containerID="b8ef5ad6fae7341669681d6901c1f6737ffc6ddbd486b850a0f70d1cf6845702"
Oct 30 19:00:20.327176 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.327056    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:20.332519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.332495    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889780-ghq6g"]
Oct 30 19:00:20.458926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.456779    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:20.473945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.473306    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:20.592791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.592132    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:20.651591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.651564    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\") pod \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") "
Oct 30 19:00:20.651742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.651674    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\" (UniqueName: \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\") pod \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\" (UID: \"c607ebcf-0806-41dd-90da-2a7b93dd5d3b\") "
Oct 30 19:00:20.651910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.651887    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca" pod "c607ebcf-0806-41dd-90da-2a7b93dd5d3b" (UID: "c607ebcf-0806-41dd-90da-2a7b93dd5d3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.652197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.652182    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.654061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.654040    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc" pod "c607ebcf-0806-41dd-90da-2a7b93dd5d3b" (UID: "c607ebcf-0806-41dd-90da-2a7b93dd5d3b"). InnerVolumeSpecName "kube-api-access-r4fnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.754126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.754047    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4fnc\" (UniqueName: \"kubernetes.io/projected/c607ebcf-0806-41dd-90da-2a7b93dd5d3b-kube-api-access-r4fnc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.764530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.764506    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:20.767921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.767905    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:20.854722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.854695    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\") pod \"43786fbc-16d8-40b2-8859-798c8897f6b3\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") "
Oct 30 19:00:20.854904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.854735    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\" (UniqueName: \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\") pod \"43786fbc-16d8-40b2-8859-798c8897f6b3\" (UID: \"43786fbc-16d8-40b2-8859-798c8897f6b3\") "
Oct 30 19:00:20.854904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.854816    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\" (UniqueName: \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\") pod \"c31e12ab-feee-4579-a928-d98ae54db325\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") "
Oct 30 19:00:20.855040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.854933    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\") pod \"c31e12ab-feee-4579-a928-d98ae54db325\" (UID: \"c31e12ab-feee-4579-a928-d98ae54db325\") "
Oct 30 19:00:20.855123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.855067    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca" pod "43786fbc-16d8-40b2-8859-798c8897f6b3" (UID: "43786fbc-16d8-40b2-8859-798c8897f6b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.855389 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.855361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca" pod "c31e12ab-feee-4579-a928-d98ae54db325" (UID: "c31e12ab-feee-4579-a928-d98ae54db325"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.855741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.855704    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43786fbc-16d8-40b2-8859-798c8897f6b3-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.855741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.855727    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e12ab-feee-4579-a928-d98ae54db325-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.857237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.857215    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x" pod "c31e12ab-feee-4579-a928-d98ae54db325" (UID: "c31e12ab-feee-4579-a928-d98ae54db325"). InnerVolumeSpecName "kube-api-access-khq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.857358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.857245    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd" pod "43786fbc-16d8-40b2-8859-798c8897f6b3" (UID: "43786fbc-16d8-40b2-8859-798c8897f6b3"). InnerVolumeSpecName "kube-api-access-dpxtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.956782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.956756    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq7x\" (UniqueName: \"kubernetes.io/projected/c31e12ab-feee-4579-a928-d98ae54db325-kube-api-access-khq7x\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.956782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:20.956779    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxtd\" (UniqueName: \"kubernetes.io/projected/43786fbc-16d8-40b2-8859-798c8897f6b3-kube-api-access-dpxtd\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.279345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.279310    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"
Oct 30 19:00:21.280193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.279344    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk" event={"ID":"43786fbc-16d8-40b2-8859-798c8897f6b3","Type":"ContainerDied","Data":"7d2f20068c3d68e7b8e8dbbb3dabc04db1740b0816f871f1314317c67cc6beb4"}
Oct 30 19:00:21.280193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.279405    5065 scope.go:122] "RemoveContainer" containerID="64565050724d555a526b9fd98951a3752aa96772f98adfc310fb024e0ab53e80"
Oct 30 19:00:21.282170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.282142    5065 generic.go:356] "Generic (PLEG): container finished" podID="7a319d16-b93e-473b-8ba6-36e299065035" containerID="28d822c98532ba09f5848d596e4f8b63a29b03a62ddf1d056c0b78155f832a3b" exitCode=60
Oct 30 19:00:21.282310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.282234    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerDied","Data":"28d822c98532ba09f5848d596e4f8b63a29b03a62ddf1d056c0b78155f832a3b"}
Oct 30 19:00:21.282647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.282629    5065 scope.go:122] "RemoveContainer" containerID="28d822c98532ba09f5848d596e4f8b63a29b03a62ddf1d056c0b78155f832a3b"
Oct 30 19:00:21.282965 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:21.282938    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt_odh-ai-gateway-infra(7a319d16-b93e-473b-8ba6-36e299065035)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" podUID="7a319d16-b93e-473b-8ba6-36e299065035"
Oct 30 19:00:21.286137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.286116    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw" event={"ID":"c607ebcf-0806-41dd-90da-2a7b93dd5d3b","Type":"ContainerDied","Data":"ab28e449a41657cb801cbf274c710cc3fd813515104680a89ab5468bebff2b19"}
Oct 30 19:00:21.286356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.286153    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"
Oct 30 19:00:21.287747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.287718    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n" event={"ID":"c31e12ab-feee-4579-a928-d98ae54db325","Type":"ContainerDied","Data":"304b2ba5b0aa9200ccf423a4c6cf16ef43b919b32a53f54f97b84f7c01904c19"}
Oct 30 19:00:21.287747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.287734    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"
Oct 30 19:00:21.295576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.295551    5065 scope.go:122] "RemoveContainer" containerID="f18652208ffdb1ba21ae0d13377e830f3be6acfe8f17a1d049f762fefe756a8d"
Oct 30 19:00:21.315626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.315604    5065 scope.go:122] "RemoveContainer" containerID="1ae43f651cc845c16dc53efd9ee1b39ba236efa646c2c722eae761a0a4b3572c"
Oct 30 19:00:21.319189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.319158    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:21.329949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.329924    5065 scope.go:122] "RemoveContainer" containerID="f531f8e2f185e2091d482b7f0f5ef3d878cf990fbcc56a265aa1c40dd2fc9ee9"
Oct 30 19:00:21.351828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.351802    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:21.359146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.359123    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889780-nkjqk"]
Oct 30 19:00:21.367633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.367604    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:21.373077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.373051    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:21.378115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.378077    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-92h7n"]
Oct 30 19:00:21.415842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.413763    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:21.418727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.416509    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889780-rr2pw"]
Oct 30 19:00:21.648912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.648886    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:21.678673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.678621    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\") pod \"a58f3803-937f-4fd3-b716-068f601481a8\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") "
Oct 30 19:00:21.678860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.678839    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\") pod \"a58f3803-937f-4fd3-b716-068f601481a8\" (UID: \"a58f3803-937f-4fd3-b716-068f601481a8\") "
Oct 30 19:00:21.679236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.679209    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca" pod "a58f3803-937f-4fd3-b716-068f601481a8" (UID: "a58f3803-937f-4fd3-b716-068f601481a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.679680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.679609    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58f3803-937f-4fd3-b716-068f601481a8-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.681257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.681236    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g" pod "a58f3803-937f-4fd3-b716-068f601481a8" (UID: "a58f3803-937f-4fd3-b716-068f601481a8"). InnerVolumeSpecName "kube-api-access-gjp6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.700735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.700714    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:21.781178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.781146    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\") pod \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") "
Oct 30 19:00:21.781327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.781272    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\" (UniqueName: \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\") pod \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\" (UID: \"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6\") "
Oct 30 19:00:21.781582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.781552    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca" pod "5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" (UID: "5ccb560e-b119-41f4-8409-5eb7a8cd5ca6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.782005 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.781982    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjp6g\" (UniqueName: \"kubernetes.io/projected/a58f3803-937f-4fd3-b716-068f601481a8-kube-api-access-gjp6g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.782289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.782052    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.783745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.783719    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6" pod "5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" (UID: "5ccb560e-b119-41f4-8409-5eb7a8cd5ca6"). InnerVolumeSpecName "kube-api-access-mg6z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.883253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.883181    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6z6\" (UniqueName: \"kubernetes.io/projected/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6-kube-api-access-mg6z6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.893416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.893388    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" path="/var/lib/kubelet/pods/0c92e810-b3f5-4a86-aadb-0ed037558f9b/volumes"
Oct 30 19:00:21.893908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.893888    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" path="/var/lib/kubelet/pods/43786fbc-16d8-40b2-8859-798c8897f6b3/volumes"
Oct 30 19:00:21.894390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.894370    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31e12ab-feee-4579-a928-d98ae54db325" path="/var/lib/kubelet/pods/c31e12ab-feee-4579-a928-d98ae54db325/volumes"
Oct 30 19:00:21.895118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:21.895100    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" path="/var/lib/kubelet/pods/c607ebcf-0806-41dd-90da-2a7b93dd5d3b/volumes"
Oct 30 19:00:22.301219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.301188    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"
Oct 30 19:00:22.301712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.301222    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7" event={"ID":"a58f3803-937f-4fd3-b716-068f601481a8","Type":"ContainerDied","Data":"604d422c1f67b94d3acd52cc8d2adc421237a683dbe6089bd985f54906074298"}
Oct 30 19:00:22.301712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.301290    5065 scope.go:122] "RemoveContainer" containerID="4a5347964f6c9efc023b714767ae57a727862a575f7797764f8a1260fbb0d63b"
Oct 30 19:00:22.307669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.307650    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"
Oct 30 19:00:22.308304 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.307689    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z" event={"ID":"5ccb560e-b119-41f4-8409-5eb7a8cd5ca6","Type":"ContainerDied","Data":"eb6f2d289f2ad07d3c2d60b82664fd9bd067b50df9dfb783d69ef7e8a85fb89f"}
Oct 30 19:00:22.316681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.316650    5065 scope.go:122] "RemoveContainer" containerID="feab690d3c8c0aa377aa63fe0814b31b9adc282e0316d9cf60b6ad5721793335"
Oct 30 19:00:22.333171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.333149    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:22.343040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.343011    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:22.355892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.355862    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889780-v8zv7"]
Oct 30 19:00:22.375479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.375433    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:22.382300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.382274    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889780-4cq8z"]
Oct 30 19:00:22.603191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.603165    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:22.712230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.712188    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\") pod \"7a319d16-b93e-473b-8ba6-36e299065035\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") "
Oct 30 19:00:22.712411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.712334    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\" (UniqueName: \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\") pod \"7a319d16-b93e-473b-8ba6-36e299065035\" (UID: \"7a319d16-b93e-473b-8ba6-36e299065035\") "
Oct 30 19:00:22.712558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.712533    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca" pod "7a319d16-b93e-473b-8ba6-36e299065035" (UID: "7a319d16-b93e-473b-8ba6-36e299065035"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:22.713033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.713014    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a319d16-b93e-473b-8ba6-36e299065035-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.714670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.714644    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj" pod "7a319d16-b93e-473b-8ba6-36e299065035" (UID: "7a319d16-b93e-473b-8ba6-36e299065035"). InnerVolumeSpecName "kube-api-access-84zlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:22.814832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:22.814748    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zlj\" (UniqueName: \"kubernetes.io/projected/7a319d16-b93e-473b-8ba6-36e299065035-kube-api-access-84zlj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.059407 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:23.059369    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:00:23.351117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.348920    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt" event={"ID":"7a319d16-b93e-473b-8ba6-36e299065035","Type":"ContainerDied","Data":"f2bb0bc7d39ad00f9357a6f68b0a7cfcd9eb25c490c02b6c53b64b6d31889e99"}
Oct 30 19:00:23.351117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.348988    5065 scope.go:122] "RemoveContainer" containerID="28d822c98532ba09f5848d596e4f8b63a29b03a62ddf1d056c0b78155f832a3b"
Oct 30 19:00:23.351117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.349125    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"
Oct 30 19:00:23.414758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.414677    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:23.421293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.421266    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889780-p7zlt"]
Oct 30 19:00:23.449978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.449949    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:00:23.450916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450896    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.450916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450914    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.450916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450927    5065 cpu_manager.go:414] "containerMap: removing container" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.450916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450932    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450938    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450945    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450955    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450961    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450969    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450973    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450981    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.450987    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451003    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451010    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451018    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451025    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451033    5065 cpu_manager.go:414] "containerMap: removing container" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451040    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451054    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451062    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451074    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451080    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451099    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451103    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451119    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451123    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451129    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451133    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451138    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451142    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451151    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451155    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451163    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451167    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451177    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451181    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451194    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.451200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451198    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451313    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451321    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451327    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451334    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451339    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451345    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451351    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451357    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451362    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451368    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451372    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451378    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c607ebcf-0806-41dd-90da-2a7b93dd5d3b" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451384    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451391    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451396    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451403    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:23.452256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.451409    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:23.456744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.456725    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.469557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.469484    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03362d\"/\"default-dockercfg-jdcj5\""
Oct 30 19:00:23.469690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.469586    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03362d\"/\"auth-test-model-03362d-kserve-self-signed-certs\""
Oct 30 19:00:23.469810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.469792    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03362d\"/\"openshift-service-ca.crt\""
Oct 30 19:00:23.474588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.474564    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:00:23.476110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.476073    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03362d\"/\"kube-root-ca.crt\""
Oct 30 19:00:23.534946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.534912    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.535156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.535125    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.535270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.535177    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45glc\" (UniqueName: \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.535343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.535298    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.535343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.535335    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.535736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.535450    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.636985    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.637040    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45glc\" (UniqueName: \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.637174    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.637209    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.637315    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.637645    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.638183    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.638789    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.659301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.639355    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.692488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.692407    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.693409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.693360    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.706749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.706723    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45glc\" (UniqueName: \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\") pod \"auth-test-model-03362d-kserve-6c4df564fc-qt5kr\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") " pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.773695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.773388    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:23.897284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.897252    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" path="/var/lib/kubelet/pods/5ccb560e-b119-41f4-8409-5eb7a8cd5ca6/volumes"
Oct 30 19:00:23.898803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.898769    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a319d16-b93e-473b-8ba6-36e299065035" path="/var/lib/kubelet/pods/7a319d16-b93e-473b-8ba6-36e299065035/volumes"
Oct 30 19:00:23.899737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:23.899710    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58f3803-937f-4fd3-b716-068f601481a8" path="/var/lib/kubelet/pods/a58f3803-937f-4fd3-b716-068f601481a8/volumes"
Oct 30 19:00:24.153150 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:00:24.153118    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10a5317_02a3_4949_a47b_f2532542c40d.slice/crio-430ffeaac5290b789e59ca00c0fa3bb97e4d655750227b046a56307f3ccb0ac2 WatchSource:0}: Error finding container 430ffeaac5290b789e59ca00c0fa3bb97e4d655750227b046a56307f3ccb0ac2: Status 404 returned error can't find the container with id 430ffeaac5290b789e59ca00c0fa3bb97e4d655750227b046a56307f3ccb0ac2
Oct 30 19:00:24.154668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:24.154639    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:00:24.374647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:24.374614    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" event={"ID":"e10a5317-02a3-4949-a47b-f2532542c40d","Type":"ContainerStarted","Data":"e5b5788c670778a7726b4a53e73516674ecaed57a7be1112d45c56a20bd9e8f5"}
Oct 30 19:00:24.375140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:24.374659    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" event={"ID":"e10a5317-02a3-4949-a47b-f2532542c40d","Type":"ContainerStarted","Data":"430ffeaac5290b789e59ca00c0fa3bb97e4d655750227b046a56307f3ccb0ac2"}
Oct 30 19:00:24.432399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:24.432301    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" podStartSLOduration=1.432287039 podStartE2EDuration="1.432287039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:24.42846404 +0000 UTC m=+1644.691928990" watchObservedRunningTime="2026-10-30 19:00:24.432287039 +0000 UTC m=+1644.695752011"
Oct 30 19:00:24.775030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:24.775002    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:25.841434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.841400    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:00:25.842180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842165    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842181    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842202    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842207    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842216    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842221    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842246    5065 cpu_manager.go:414] "containerMap: removing container" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842250    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:25.842256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842261    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842265    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842361    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf4c13-25c3-4f09-a7af-a8b287595fac" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842370    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e12ab-feee-4579-a928-d98ae54db325" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842378    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58f3803-937f-4fd3-b716-068f601481a8" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842384    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccb560e-b119-41f4-8409-5eb7a8cd5ca6" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842393    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92e810-b3f5-4a86-aadb-0ed037558f9b" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842399    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a319d16-b93e-473b-8ba6-36e299065035" containerName="cleanup"
Oct 30 19:00:25.842560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.842403    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="43786fbc-16d8-40b2-8859-798c8897f6b3" containerName="cleanup"
Oct 30 19:00:25.846608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.846582    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.852584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.852560    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0a962a-openshift-default-dockercfg-msm6z\""
Oct 30 19:00:25.870079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.870054    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:00:25.960946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.960911    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.960954    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961023    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961123    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961151    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84sc\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961226    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961283    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961369    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:25.961500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:25.961446    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.077119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.074795    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.077119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.074839    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.077119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.074918    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.077119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.075002    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.077119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.075029    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84sc\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.107252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.100783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.107252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.101675    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.107252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.106415    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.108167    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.108271    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.108404    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.108501    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.108901    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.109140    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.111109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.109978    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.130112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.125452    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.158124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.157929    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.172111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.166938    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84sc\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:26.172111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:26.167228    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:27.431225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:27.431177    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:27.712336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:27.703743    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:27.712336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:27.709187    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:27.712336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:27.709260    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:27.795111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:27.789356    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:00:28.049383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.049338    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:28.228352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.223320    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:28.334170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.333867    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:28.431395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.431354    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:28.451474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.451432    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:28.452695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.452667    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:00:28.479685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.479651    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:28.666172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.582910    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:00:28.666172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:28.588373    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:29.169874 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.167929    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:29.192109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.189209    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" event={"ID":"ac748607-954a-4ae1-9803-467c3458b486","Type":"ContainerStarted","Data":"4cff79c2f6dfbc1cb7ec154c1cc11053e9a76496a16e2a0df77ad81b7359aea7"}
Oct 30 19:00:29.192109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.189323    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:00:29.401173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.396360    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:29.428961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.401935    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:00:29.511114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.495664    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:29.512402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.512377    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:00:29.710167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.687018    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:29.710167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:29.688981    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:00:30.204008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.169256    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 19:00:30.210493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.210458    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:00:30.352146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.348538    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:30.409233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.409192    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:30.535342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.535300    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:00:30.559112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.555870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" event={"ID":"ac748607-954a-4ae1-9803-467c3458b486","Type":"ContainerStarted","Data":"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"}
Oct 30 19:00:30.565443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.565385    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podStartSLOduration=5.565364033 podStartE2EDuration="5.565364033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:30.493538359 +0000 UTC m=+1650.757003308" watchObservedRunningTime="2026-10-30 19:00:30.565364033 +0000 UTC m=+1650.828828984"
Oct 30 19:00:30.688289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.688242    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:00:30.688744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:30.688717    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:31.172441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.172406    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:31.172815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.172789    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:31.172969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.172946    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:31.427178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.426412    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.427178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.427036    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.460109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.428716    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.460109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.429169    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:00:31.460109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.430120    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.558982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.548657    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.581336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.581293    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:31.691173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.689695    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:00:31.849172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.828353    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:31.870137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.849954    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:00:31.870137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:31.850552    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:00:32.219174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.180944    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:32.219174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.180998    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:32.436296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.436247    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:32.437842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.437794    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:00:32.450149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.444327    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:00:32.450149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.448351    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:00:32.594950 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.594905    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:32.744113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.741219    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:32.865632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.863271    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:32.865632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:32.863560    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:33.180984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.180890    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:33.181392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.181355    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:33.328481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.328445    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:00:33.439370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.439270    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:33.454144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.453324    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:33.605109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.604869    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:33.846141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.843514    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:33.874589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.874550    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:33.896243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.874831    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:33.975137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:33.954549    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.180533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.180447    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:34.180798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.180767    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:34.455034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.454936    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.469263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.469228    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.513567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.469876    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.608189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.607168    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.701278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.701227    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 19:00:34.851146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.846165    5065 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.255:8000/health\": context deadline exceeded"
Oct 30 19:00:34.872306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.871495    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:00:34.892156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.883165    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.004175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:34.997329    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:00:35.055860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:35.038536    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:00:35.055860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:35.047593    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:00:35.061143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:35.060488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:00:35.178772 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:35.175475    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:35.178772 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:35.175529    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:36.169902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:36.169440    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:36.169902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:36.169499    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:36.932111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:36.881148    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:00:37.037472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.037433    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.069501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.069477    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133165    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133224    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133351    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133387    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133457    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133510    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5l7\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133540    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133675    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.135512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.133708    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.169265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.169023    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:00:37.169265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.169118    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 19:00:37.235195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235069    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235164    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235286    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235322    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235389    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235539    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5l7\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235573    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.235727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235716    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.236052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235754    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.236052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.235792    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.236854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.236830    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.237649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.237490    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.237781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.237657    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.237994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.237914    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.239688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.239649    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.240187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.240160    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.253503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.253470    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.254057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.254021    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5l7\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.372278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.371992    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:37.895676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.882604    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:37.895676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.882705    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:37.895676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.882752    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:37.967947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:37.967913    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:00:38.272558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.271996    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" event={"ID":"6337fbdf-bb56-480e-adea-4f20ca6b2a5f","Type":"ContainerStarted","Data":"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"}
Oct 30 19:00:38.272558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.272065    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" event={"ID":"6337fbdf-bb56-480e-adea-4f20ca6b2a5f","Type":"ContainerStarted","Data":"22e9c6029f89b89bcbb0c88cdb788ef4a7718a0d3283e240e546bb298d0c94c9"}
Oct 30 19:00:38.338445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.338364    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podStartSLOduration=2.338343515 podStartE2EDuration="2.338343515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.336221931 +0000 UTC m=+1658.599686880" watchObservedRunningTime="2026-10-30 19:00:38.338343515 +0000 UTC m=+1658.601808465"
Oct 30 19:00:38.359559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.358220    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:38.359559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.359169    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:38.360450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.360425    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:00:38.372889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.372673    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:38.530630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.530510    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:38.530902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.530872    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:38.625387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.625297    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 19:00:38.626919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.625980    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy" containerID="cri-o://920d135993d43415e3069f29532bc8d4189279dbaf4eac5a4b64131f43bf1a4d" gracePeriod=30
Oct 30 19:00:38.767118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.766257    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:38.767118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:38.766323    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.374884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:39.374855    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:39.375427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:39.374903    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:40.372977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:40.372868    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:40.372977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:40.372936    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:40.758309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:40.758233    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:40.758675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:40.758303    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:41.373330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:41.373298    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:41.373524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:41.373353    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:42.373115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:42.373067    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:42.373615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:42.373153    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:43.373604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:43.373562    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:43.374005 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:43.373643    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:43.870024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:43.868635    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:44.373579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.373500    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 19:00:44.373579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.373561    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 19:00:44.410642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.410610    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:00:44.411028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.411010    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"]
Oct 30 19:00:44.430785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.430079    5065 generic.go:356] "Generic (PLEG): container finished" podID="1ae936fb-f27b-4220-bee1-d985c650d875" containerID="920d135993d43415e3069f29532bc8d4189279dbaf4eac5a4b64131f43bf1a4d" exitCode=0
Oct 30 19:00:44.507467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.484544    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"]
Oct 30 19:00:44.507467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.484590    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" event={"ID":"1ae936fb-f27b-4220-bee1-d985c650d875","Type":"ContainerDied","Data":"920d135993d43415e3069f29532bc8d4189279dbaf4eac5a4b64131f43bf1a4d"}
Oct 30 19:00:44.507467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.484698    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.544161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.525940    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:00:44.544161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.526621    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:44.576128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.575819    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.651347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.645678    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8sx\" (UniqueName: \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.651347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.645931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.742334 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.742302    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 19:00:44.747125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747073    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747151    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747186    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747252    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747276    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747348    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747441    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747494    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.747789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.747769    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\") pod \"1ae936fb-f27b-4220-bee1-d985c650d875\" (UID: \"1ae936fb-f27b-4220-bee1-d985c650d875\") "
Oct 30 19:00:44.748063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.748032    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.748311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.748287    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.748401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.748360    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8sx\" (UniqueName: \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.749546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.749442    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:44.750374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.750330    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:44.751380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.750524    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:44.751380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.751204    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.751992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.751959    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:44.752279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.752248    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:44.757599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.756965    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.763249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.763220    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:44.763767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.763744    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:44.767290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.767263    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "kube-api-access-dzwnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:44.767501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.767464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo" pod "1ae936fb-f27b-4220-bee1-d985c650d875" (UID: "1ae936fb-f27b-4220-bee1-d985c650d875"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:00:44.857099 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.848747    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8sx\" (UniqueName: \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\") pod \"maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859061    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859156    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859177    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859194    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859208    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae936fb-f27b-4220-bee1-d985c650d875-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859222    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859236    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae936fb-f27b-4220-bee1-d985c650d875-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859252    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae936fb-f27b-4220-bee1-d985c650d875-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.859340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:44.859265    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwnk\" (UniqueName: \"kubernetes.io/projected/1ae936fb-f27b-4220-bee1-d985c650d875-kube-api-access-dzwnk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:45.028375 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.028340    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:45.394467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.392887    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:45.394467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.394363    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:45.396518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.396460    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:00:45.655170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.652786    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"
Oct 30 19:00:45.655170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.653333    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj" event={"ID":"1ae936fb-f27b-4220-bee1-d985c650d875","Type":"ContainerDied","Data":"12351dceaf880209a1dcaa0c4beea3c06ff58a71476ccc92416e5251ee436d99"}
Oct 30 19:00:45.655170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.653444    5065 scope.go:122] "RemoveContainer" containerID="920d135993d43415e3069f29532bc8d4189279dbaf4eac5a4b64131f43bf1a4d"
Oct 30 19:00:45.793697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.783735    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"]
Oct 30 19:00:45.816564 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.816477    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 19:00:45.853605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.853567    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kwwdj"]
Oct 30 19:00:45.947695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:45.947665    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" path="/var/lib/kubelet/pods/1ae936fb-f27b-4220-bee1-d985c650d875/volumes"
Oct 30 19:00:46.676395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:46.676300    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v" event={"ID":"f124b147-2de5-4602-b235-dc1b54f22e1b","Type":"ContainerStarted","Data":"2dd7b56022fbbed7b3b9b2a977d98e1d9a6f17a35d9ccf1a5530899175d8c441"}
Oct 30 19:00:46.676395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:46.676364    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v" event={"ID":"f124b147-2de5-4602-b235-dc1b54f22e1b","Type":"ContainerStarted","Data":"377e0eba776dc6d0d7fa3f1fff9decba0a55f9833dfa256b7aa477bec28c8743"}
Oct 30 19:00:46.706249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:46.706187    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v" podStartSLOduration=2.706165415 podStartE2EDuration="2.706165415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:46.698363117 +0000 UTC m=+1666.961828067" watchObservedRunningTime="2026-10-30 19:00:46.706165415 +0000 UTC m=+1666.969630363"
Oct 30 19:00:47.668999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.668921    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:00:47.673498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.673473    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy"
Oct 30 19:00:47.736821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.736787    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy"
Oct 30 19:00:47.737393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.737071    5065 generic.go:356] "Generic (PLEG): container finished" podID="f124b147-2de5-4602-b235-dc1b54f22e1b" containerID="2dd7b56022fbbed7b3b9b2a977d98e1d9a6f17a35d9ccf1a5530899175d8c441" exitCode=0
Oct 30 19:00:47.739452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.739427    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae936fb-f27b-4220-bee1-d985c650d875" containerName="istio-proxy"
Oct 30 19:00:47.755269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.755226    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v" event={"ID":"f124b147-2de5-4602-b235-dc1b54f22e1b","Type":"ContainerDied","Data":"2dd7b56022fbbed7b3b9b2a977d98e1d9a6f17a35d9ccf1a5530899175d8c441"}
Oct 30 19:00:47.755438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.755296    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:00:47.755438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.755403    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.762544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.762417    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c3a5b9\"/\"kube-root-ca.crt\""
Oct 30 19:00:47.762544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.762493    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c3a5b9\"/\"openshift-service-ca.crt\""
Oct 30 19:00:47.766536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.766511    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c3a5b9\"/\"auth-test-model-c3a5b9-kserve-self-signed-certs\""
Oct 30 19:00:47.766824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.766788    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c3a5b9\"/\"default-dockercfg-f8mrf\""
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823331    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gc2l\" (UniqueName: \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823442    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823530    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823573    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823635    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.823911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.823665    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.942886    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gc2l\" (UniqueName: \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943014    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943129    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943173    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943239    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943272    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.943880    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.944725    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.946159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.945281    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:47.984110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:47.963040    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:48.072172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.051186    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:48.072172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.057953    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gc2l\" (UniqueName: \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\") pod \"auth-test-model-c3a5b9-kserve-5587bf88d5-g8775\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") " pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:48.121621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.121341    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:48.533646 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:00:48.533588    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b950233_52f8_4bb4_99dd_6760d95bc6a6.slice/crio-205af4728dd36829323b1b24f01365e59cdf6c6ac2e82e15b8f403074e5e65e0 WatchSource:0}: Error finding container 205af4728dd36829323b1b24f01365e59cdf6c6ac2e82e15b8f403074e5e65e0: Status 404 returned error can't find the container with id 205af4728dd36829323b1b24f01365e59cdf6c6ac2e82e15b8f403074e5e65e0
Oct 30 19:00:48.536473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.536203    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:00:48.760857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.760767    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" event={"ID":"9b950233-52f8-4bb4-99dd-6760d95bc6a6","Type":"ContainerStarted","Data":"2f6b1a8ea8651a0e1c2ee1c041d8ac7619d27d64d7d7b741e1dd0dbaf27d82d2"}
Oct 30 19:00:48.760857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.760817    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" event={"ID":"9b950233-52f8-4bb4-99dd-6760d95bc6a6","Type":"ContainerStarted","Data":"205af4728dd36829323b1b24f01365e59cdf6c6ac2e82e15b8f403074e5e65e0"}
Oct 30 19:00:48.810497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:48.810399    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" podStartSLOduration=1.81038418 podStartE2EDuration="1.81038418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:48.802740338 +0000 UTC m=+1669.066205287" watchObservedRunningTime="2026-10-30 19:00:48.81038418 +0000 UTC m=+1669.073849127"
Oct 30 19:00:49.048300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.048220    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"]
Oct 30 19:00:49.061126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.059512    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f973f20-jlmjp"]
Oct 30 19:00:49.125367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.125306    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:49.148377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.145607    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:49.216136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.214821    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\") pod \"f124b147-2de5-4602-b235-dc1b54f22e1b\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") "
Oct 30 19:00:49.216136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.214971    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\") pod \"f124b147-2de5-4602-b235-dc1b54f22e1b\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") "
Oct 30 19:00:49.216136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.215119    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\" (UniqueName: \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\") pod \"f124b147-2de5-4602-b235-dc1b54f22e1b\" (UID: \"f124b147-2de5-4602-b235-dc1b54f22e1b\") "
Oct 30 19:00:49.216902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.216858    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca" pod "f124b147-2de5-4602-b235-dc1b54f22e1b" (UID: "f124b147-2de5-4602-b235-dc1b54f22e1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:49.219303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.219277    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca" pod "f124b147-2de5-4602-b235-dc1b54f22e1b" (UID: "f124b147-2de5-4602-b235-dc1b54f22e1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:49.221619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.221520    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx" pod "f124b147-2de5-4602-b235-dc1b54f22e1b" (UID: "f124b147-2de5-4602-b235-dc1b54f22e1b"). InnerVolumeSpecName "kube-api-access-tp8sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:49.318120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.318000    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp8sx\" (UniqueName: \"kubernetes.io/projected/f124b147-2de5-4602-b235-dc1b54f22e1b-kube-api-access-tp8sx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.318120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.318038    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.318120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.318052    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f124b147-2de5-4602-b235-dc1b54f22e1b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.774214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.774185    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"
Oct 30 19:00:49.774639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.774215    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v" event={"ID":"f124b147-2de5-4602-b235-dc1b54f22e1b","Type":"ContainerDied","Data":"377e0eba776dc6d0d7fa3f1fff9decba0a55f9833dfa256b7aa477bec28c8743"}
Oct 30 19:00:49.774639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.774284    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377e0eba776dc6d0d7fa3f1fff9decba0a55f9833dfa256b7aa477bec28c8743"
Oct 30 19:00:49.915046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:49.915014    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed24ee1a-c381-4735-b749-34a5dcdf5619" path="/var/lib/kubelet/pods/ed24ee1a-c381-4735-b749-34a5dcdf5619/volumes"
Oct 30 19:00:53.120560 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:00:53.120520    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:00:53.494361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.484282    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:53.494361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.484323    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.666575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.666396    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": read tcp 10.132.0.2:58268->10.132.0.242:15021: read: connection reset by peer" start-of-body=
Oct 30 19:00:53.666575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.666463    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": read tcp 10.132.0.2:58268->10.132.0.242:15021: read: connection reset by peer"
Oct 30 19:00:53.835780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.835749    5065 generic.go:356] "Generic (PLEG): container finished" podID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerID="fdb6b654d4a5d264741fa9e0e275fb190f0b46c14349b012f969e28f53353664" exitCode=0
Oct 30 19:00:53.835966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.835818    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerDied","Data":"fdb6b654d4a5d264741fa9e0e275fb190f0b46c14349b012f969e28f53353664"}
Oct 30 19:00:53.836379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.836358    5065 scope.go:122] "RemoveContainer" containerID="fdb6b654d4a5d264741fa9e0e275fb190f0b46c14349b012f969e28f53353664"
Oct 30 19:00:53.837940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.837918    5065 generic.go:356] "Generic (PLEG): container finished" podID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerID="80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83" exitCode=0
Oct 30 19:00:53.838062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.837986    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerDied","Data":"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"}
Oct 30 19:00:53.838182 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.838130    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.838254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.838227    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.838335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.838315    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.838402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.838348    5065 scope.go:122] "RemoveContainer" containerID="80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"
Oct 30 19:00:53.839971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.839945    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.840146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.840020    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.840146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.840069    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.841000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.840970    5065 generic.go:356] "Generic (PLEG): container finished" podID="229795d4-f03f-4940-a980-59606344d520" containerID="a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520" exitCode=0
Oct 30 19:00:53.841149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.841014    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerDied","Data":"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"}
Oct 30 19:00:53.841412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.841397    5065 scope.go:122] "RemoveContainer" containerID="a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"
Oct 30 19:00:53.850720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.850681    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.850830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.850742    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:53.850830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:53.850773    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:00:54.339947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:54.339799    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 19:00:54.718158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:54.717973    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:00:54.912107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:54.912027    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerStarted","Data":"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"}
Oct 30 19:00:54.929754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:54.929677    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerStarted","Data":"b44e4e6ec20240115d2254c9374d694bf6bfae83dce1dc7e0d5e3113ac8ef9ed"}
Oct 30 19:00:54.936152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:54.933154    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerStarted","Data":"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"}
Oct 30 19:00:55.006004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.005904    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:00:55.006624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.006594    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.006752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.006645    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:00:55.125803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.125767    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:00:55.126003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.125953    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.126137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.126020    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:00:55.669117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.659687    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:00:55.669117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.659857    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.669117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:55.659916    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:00:56.006344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.006260    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.006615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.006581    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:00:56.133177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.131649    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.133177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.131738    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:00:56.676364 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.676328    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.676972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:56.676940    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:00:57.019860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.006601    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:00:57.019860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.006675    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:00:57.158359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.152177    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:00:57.158713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.158627    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:00:57.669154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.658206    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:00:57.669154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:57.658284    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:00:58.014110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.011772    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.014110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.011832    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:00:58.132227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.132177    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.132428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.132239    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:00:58.156113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.153758    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:58.342168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.341488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:58.649174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.648820    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.649174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:58.648880    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:00:59.111196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.111155    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.112020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.111979    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:00:59.189286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.189250    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.189564 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.189534    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:00:59.313118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.304497    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:00:59.344112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.343332    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" podUID="708cce75-1b42-4ecc-9de0-abb9853d7a28" containerName="payload-processing" containerID="cri-o://94271e9e3bfb232d715b085bbbe3de5ed42bdffac7fe0f2d42e288ff7c3e84a1" gracePeriod=25
Oct 30 19:00:59.362373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.362288    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"]
Oct 30 19:00:59.568150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.561025    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9872e69a-rt5m4"]
Oct 30 19:00:59.665167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.654188    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.692022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:00:59.691975    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:01:00.462343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.462286    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.472418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.472382    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:00.896109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.892467    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.896109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.892520    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:01:00.896109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.895414    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e93cf41-7919-4197-8fdc-ea8b8ed53310" path="/var/lib/kubelet/pods/0e93cf41-7919-4197-8fdc-ea8b8ed53310/volumes"
Oct 30 19:01:00.896109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.895903    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:01:00.924629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.916339    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f124b147-2de5-4602-b235-dc1b54f22e1b" containerName="revoke-keys"
Oct 30 19:01:00.924629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.916367    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f124b147-2de5-4602-b235-dc1b54f22e1b" containerName="revoke-keys"
Oct 30 19:01:00.924629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.916521    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f124b147-2de5-4602-b235-dc1b54f22e1b" containerName="revoke-keys"
Oct 30 19:01:00.924629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.921580    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.924629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.921622    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.940172    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.941884    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.941955    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.941984    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942005    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942070    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942120    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942184    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942219    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqkm\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.954126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.942393    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:00.980703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.975335    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.980703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.975407    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:01:00.996292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:00.996248    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:01.041136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.035681    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-86c150-openshift-default-dockercfg-htf9s\""
Oct 30 19:01:01.042327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.042301    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:01:01.042525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.042503    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:01:01.072377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.072337    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:01.073716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.072515    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:01:01.126300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.126267    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 19:01:01.126569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.126542    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 19:01:01.139402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.139363    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.152422    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqkm\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.152745    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.152859    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.152965    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.153022    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.153056    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.153194    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.161163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.153235    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.181712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.181678    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.204378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.204346    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.204849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.204827    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.205579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.205555    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.209440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.209419    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.229529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.229500    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqkm\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.384472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.336756    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.441172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.402450    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.441172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.402711    5065 generic.go:356] "Generic (PLEG): container finished" podID="708cce75-1b42-4ecc-9de0-abb9853d7a28" containerID="94271e9e3bfb232d715b085bbbe3de5ed42bdffac7fe0f2d42e288ff7c3e84a1" exitCode=0
Oct 30 19:01:01.441172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.413306    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 19:01:01.441172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.413721    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" event={"ID":"708cce75-1b42-4ecc-9de0-abb9853d7a28","Type":"ContainerDied","Data":"94271e9e3bfb232d715b085bbbe3de5ed42bdffac7fe0f2d42e288ff7c3e84a1"}
Oct 30 19:01:01.441172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.413792    5065 scope.go:122] "RemoveContainer" containerID="94271e9e3bfb232d715b085bbbe3de5ed42bdffac7fe0f2d42e288ff7c3e84a1"
Oct 30 19:01:01.500207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.499477    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.653191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.634611    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\" (UniqueName: \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\") pod \"708cce75-1b42-4ecc-9de0-abb9853d7a28\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") "
Oct 30 19:01:01.653191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.634730    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\") pod \"708cce75-1b42-4ecc-9de0-abb9853d7a28\" (UID: \"708cce75-1b42-4ecc-9de0-abb9853d7a28\") "
Oct 30 19:01:01.659632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.659604    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:01:01.659853 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.659826    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:01:01.661095 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.661041    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume" pod "708cce75-1b42-4ecc-9de0-abb9853d7a28" (UID: "708cce75-1b42-4ecc-9de0-abb9853d7a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:01.665544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.665515    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4" pod "708cce75-1b42-4ecc-9de0-abb9853d7a28" (UID: "708cce75-1b42-4ecc-9de0-abb9853d7a28"). InnerVolumeSpecName "kube-api-access-kclg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:01.707468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.707376    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:01.741801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.741766    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclg4\" (UniqueName: \"kubernetes.io/projected/708cce75-1b42-4ecc-9de0-abb9853d7a28-kube-api-access-kclg4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:01.742041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:01.742021    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708cce75-1b42-4ecc-9de0-abb9853d7a28-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:02.077565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.077514    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:01:02.078175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.078153    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"]
Oct 30 19:01:02.081170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.081153    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:01:02.081361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.081348    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7307a27a-sd7hz"]
Oct 30 19:01:02.115109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.089889    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:01:02.175545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.172831    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:01:02.175545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.174531    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:01:02.178613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.178584    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:01:02.220951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.220920    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:01:02.226113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.226047    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:02.226386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.226361    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:02.226537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.226519    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:02.415880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.415843    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" event={"ID":"51f029b4-0921-4f4a-bc47-25d2920bf00f","Type":"ContainerStarted","Data":"2ed9a1ee27eb30c83ed26ded5c51ae29b610a40061c5b037770189f20b7d9fdb"}
Oct 30 19:01:02.416129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.415896    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" event={"ID":"51f029b4-0921-4f4a-bc47-25d2920bf00f","Type":"ContainerStarted","Data":"bfca57e4144c5bdcdadb6ddd573a9f287c4210b267580396e45821a70d6edced"}
Oct 30 19:01:02.418614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.418590    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"
Oct 30 19:01:02.419028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.419012    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p" event={"ID":"708cce75-1b42-4ecc-9de0-abb9853d7a28","Type":"ContainerDied","Data":"185a5658bca8e5ffcd59adc94f0e3533ab4bc1aa8dff87e86aef42163ffe46c5"}
Oct 30 19:01:02.444110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.444042    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podStartSLOduration=3.444026977 podStartE2EDuration="3.444026977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:02.442816076 +0000 UTC m=+1682.706281046" watchObservedRunningTime="2026-10-30 19:01:02.444026977 +0000 UTC m=+1682.707491926"
Oct 30 19:01:02.465759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.465672    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 19:01:02.470304 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.470277    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-743e0b-57f976c6d-fqw2p"]
Oct 30 19:01:02.653896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.653869    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:01:02.654369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.654350    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:01:02.655096 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.655063    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:01:02.708544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:02.708512    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:03.716238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:03.716196    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:03.716854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:03.716264    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:03.977780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:03.977695    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2701c4-0c1d-4640-87a6-87da13ba67d5" path="/var/lib/kubelet/pods/5d2701c4-0c1d-4640-87a6-87da13ba67d5/volumes"
Oct 30 19:01:03.978189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:03.978173    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708cce75-1b42-4ecc-9de0-abb9853d7a28" path="/var/lib/kubelet/pods/708cce75-1b42-4ecc-9de0-abb9853d7a28/volumes"
Oct 30 19:01:04.708443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:04.708404    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:04.708717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:04.708689    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:05.759111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:05.709307    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:05.759111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:05.709376    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:06.711246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:06.711212    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:06.717468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:06.717432    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:07.711114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:07.710982    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:07.711114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:07.711065    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:08.524148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:08.521161    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"]
Oct 30 19:01:08.709820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:08.706637    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-743e0b-fcz5v"]
Oct 30 19:01:08.711434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:08.711404    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:08.711990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:08.711959    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:09.730110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:09.709521    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Oct 30 19:01:09.730110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:09.709590    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Oct 30 19:01:09.898779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:09.898740    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f124b147-2de5-4602-b235-dc1b54f22e1b" path="/var/lib/kubelet/pods/f124b147-2de5-4602-b235-dc1b54f22e1b/volumes"
Oct 30 19:01:10.363606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.363572    5065 scope.go:122] "RemoveContainer" containerID="0880ecae894aeec6e162d65f80b5466cb3f6ddf2ddd9404005b4c842b9ca6558"
Oct 30 19:01:10.375378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.375341    5065 scope.go:122] "RemoveContainer" containerID="15f4937d6db41568b5403243b2b4e5b620eda510c4962363dabf7b26334604be"
Oct 30 19:01:10.389270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.389243    5065 scope.go:122] "RemoveContainer" containerID="a74acb8b0f6061bd89c311f001dc2bfb78ff02f7bbbbe2c9e4842cc87784eaac"
Oct 30 19:01:10.713761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.713673    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:10.714229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.714203    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:10.714852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:10.714834    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:01:11.035925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:11.035893    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"]
Oct 30 19:01:11.042333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:11.042308    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c2c241-wtghx"]
Oct 30 19:01:11.924608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:11.924575    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0d9fe9-a075-46a1-9f49-c974594d8d52" path="/var/lib/kubelet/pods/3e0d9fe9-a075-46a1-9f49-c974594d8d52/volumes"
Oct 30 19:01:16.048581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.048549    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"]
Oct 30 19:01:16.053849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.053822    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"]
Oct 30 19:01:16.057969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.057949    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d684db0-xd5wj"]
Oct 30 19:01:16.071286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.069593    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8c28ff-h79rx"]
Oct 30 19:01:16.631632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.631598    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 19:01:16.632227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:16.632169    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" containerID="cri-o://b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970" gracePeriod=30
Oct 30 19:01:17.656566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.656528    5065 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:01:17.656998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.656594    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:01:17.758146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.758107    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:17.759628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.759501    5065 cpu_manager.go:414] "containerMap: removing container" podUID="708cce75-1b42-4ecc-9de0-abb9853d7a28" containerName="payload-processing"
Oct 30 19:01:17.759774 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.759662    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708cce75-1b42-4ecc-9de0-abb9853d7a28" containerName="payload-processing"
Oct 30 19:01:17.759988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.759971    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="708cce75-1b42-4ecc-9de0-abb9853d7a28" containerName="payload-processing"
Oct 30 19:01:17.767017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.766992    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.779942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.779914    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-aec752-openshift-default-dockercfg-rpbrw\""
Oct 30 19:01:17.786469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.786438    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:17.831395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831362    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831409    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831460    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831475    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4c7\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831492    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831566    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831900 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831608    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831900 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831676    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.831900 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.831697    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.936393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.936305    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.941123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.941078    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.945285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.945258    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.945468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.945451    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4c7\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.984202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.984165    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:17.989839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.984572    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.000924    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.001059    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.001122    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.001946    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.941963    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.940829    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e4bc45-1c84-421b-b763-3d28f2a6bdac" path="/var/lib/kubelet/pods/86e4bc45-1c84-421b-b763-3d28f2a6bdac/volumes"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.010423    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2e397e-6167-49c5-a230-ba39ba720504" path="/var/lib/kubelet/pods/9f2e397e-6167-49c5-a230-ba39ba720504/volumes"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.010712    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.968615    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:17.942377    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.010960    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.024148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.023740    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4c7\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.045754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.027363    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.088651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.064213    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\") pod \"e2e-noten-aec752-openshift-default-587f54fb87-qs6ph\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") " pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.088651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.088393    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:18.537800 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:01:18.537774    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff4d114_bb4c_4f1b_85c1_9fcddc6af4b5.slice/crio-6a9c6c27ddaa478e47ace1650878d690b7cba97dc473be0b5f9b3c4a248e7eea WatchSource:0}: Error finding container 6a9c6c27ddaa478e47ace1650878d690b7cba97dc473be0b5f9b3c4a248e7eea: Status 404 returned error can't find the container with id 6a9c6c27ddaa478e47ace1650878d690b7cba97dc473be0b5f9b3c4a248e7eea
Oct 30 19:01:18.539892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.539815    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:18.540833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.540800    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:18.540919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.540864    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:18.540919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:18.540891    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:19.035427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:19.035379    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" event={"ID":"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5","Type":"ContainerStarted","Data":"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"}
Oct 30 19:01:19.035836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:19.035482    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" event={"ID":"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5","Type":"ContainerStarted","Data":"6a9c6c27ddaa478e47ace1650878d690b7cba97dc473be0b5f9b3c4a248e7eea"}
Oct 30 19:01:19.068341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:19.068295    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" podStartSLOduration=2.06828214 podStartE2EDuration="2.06828214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:19.065630929 +0000 UTC m=+1699.329095933" watchObservedRunningTime="2026-10-30 19:01:19.06828214 +0000 UTC m=+1699.331747088"
Oct 30 19:01:19.092497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:19.092459    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:19.098478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:19.098457    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:20.045682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:20.045645    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:20.046826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:20.046807    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:21.977886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:21.977863    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:01:22.004700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.004673    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.004845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.004718    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.004845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.004797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.004975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.004856    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.004975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.004909    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.005100 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005012    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.005161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005125    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.005202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005164    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.005202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005169    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.005318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005251    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\") pod \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\" (UID: \"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13\") "
Oct 30 19:01:22.005318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005280    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:22.005434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005335    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.005533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005489    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.005663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.005642    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.006078 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.006057    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.006208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.006100    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.006208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.006119    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.006208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.006133    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.006208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.006146    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.009227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.008740    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.009227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.008758    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:22.009227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.008783    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:22.009227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.008809    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8" pod "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" (UID: "3a1b7cf8-2b0d-4437-be2d-55631e8c9f13"). InnerVolumeSpecName "kube-api-access-ltwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:22.069740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.069710    5065 generic.go:356] "Generic (PLEG): container finished" podID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerID="b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970" exitCode=0
Oct 30 19:01:22.069919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.069769    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"
Oct 30 19:01:22.069919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.069794    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerDied","Data":"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"}
Oct 30 19:01:22.069919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.069850    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht" event={"ID":"3a1b7cf8-2b0d-4437-be2d-55631e8c9f13","Type":"ContainerDied","Data":"335e0291d73d18c9e92a3f9fdbfc89218b13a1fb2569dcd7dad06376931d66e2"}
Oct 30 19:01:22.069919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.069879    5065 scope.go:122] "RemoveContainer" containerID="b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"
Oct 30 19:01:22.091829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.091805    5065 scope.go:122] "RemoveContainer" containerID="80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"
Oct 30 19:01:22.108203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.107487    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.108203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.107520    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.108203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.107537    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.108203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.107550    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwm8\" (UniqueName: \"kubernetes.io/projected/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13-kube-api-access-ltwm8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.116386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.116361    5065 scope.go:122] "RemoveContainer" containerID="b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"
Oct 30 19:01:22.117057 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:01:22.116902    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970\": container with ID starting with b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970 not found: ID does not exist" containerID="b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"
Oct 30 19:01:22.117057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.116940    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970"} err="failed to get container status \"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970\": rpc error: code = NotFound desc = could not find container \"b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970\": container with ID starting with b9fc54c09e9f93b653ba70c55caf5bd73ed3bd0bbbeb06e71179d8b436b9a970 not found: ID does not exist"
Oct 30 19:01:22.117057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.116968    5065 scope.go:122] "RemoveContainer" containerID="80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"
Oct 30 19:01:22.117415 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:01:22.117368    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83\": container with ID starting with 80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83 not found: ID does not exist" containerID="80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"
Oct 30 19:01:22.117499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.117426    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83"} err="failed to get container status \"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83\": rpc error: code = NotFound desc = could not find container \"80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83\": container with ID starting with 80e94cd9b4f80dd93eb86e934bc1f3472aa5e8a197d7bc711d0de9d8eccfdf83 not found: ID does not exist"
Oct 30 19:01:22.122246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.122200    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 19:01:22.139129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:22.139059    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-ltzht"]
Oct 30 19:01:23.059044 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:01:23.059002    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:01:23.894201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:23.894159    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" path="/var/lib/kubelet/pods/3a1b7cf8-2b0d-4437-be2d-55631e8c9f13/volumes"
Oct 30 19:01:26.038625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:26.038591    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"]
Oct 30 19:01:26.044016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:26.043986    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b094b15-xnx77"]
Oct 30 19:01:27.894053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:27.894019    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9d3be0-5437-4266-95c7-79bd7ca90a18" path="/var/lib/kubelet/pods/fe9d3be0-5437-4266-95c7-79bd7ca90a18/volumes"
Oct 30 19:01:28.044594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:28.044561    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"]
Oct 30 19:01:28.061079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:28.061041    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e789facf-zvxf9"]
Oct 30 19:01:29.906910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:29.906873    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fec6307-d9df-4ff6-a449-f99c1f082545" path="/var/lib/kubelet/pods/5fec6307-d9df-4ff6-a449-f99c1f082545/volumes"
Oct 30 19:01:31.496515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.496481    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:01:31.497704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497683    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.497787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497708    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.497787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497746    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.497787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497755    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.497950 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497937    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.497990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.497968    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1b7cf8-2b0d-4437-be2d-55631e8c9f13" containerName="istio-proxy"
Oct 30 19:01:31.503760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.503734    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.512356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.512332    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c3a5b9-openshift-default-dockercfg-b8nm9\""
Oct 30 19:01:31.517700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.517679    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:01:31.618960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.618929    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.618966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619213    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619240    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdw2\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619288    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619420    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619464    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619533    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.619609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.619595    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721212    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721246    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdw2\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721293    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721358    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721427    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721469    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721574    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721602    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721657    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.721825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721711    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.722068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721908    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.722068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.721932    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.722624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.722580    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.724188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.724163    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.724549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.724531    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.734650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.734630    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.735827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.735795    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdw2\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:31.824132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:31.824079    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:32.162521 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:01:32.162487    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e68cd6_acb9_4c00_b3bd_4bf1089cbb2a.slice/crio-f87fbeafe04ed3b09601763303287cb91c18da296680bfea7e7366642c571937 WatchSource:0}: Error finding container f87fbeafe04ed3b09601763303287cb91c18da296680bfea7e7366642c571937: Status 404 returned error can't find the container with id f87fbeafe04ed3b09601763303287cb91c18da296680bfea7e7366642c571937
Oct 30 19:01:32.163168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.163145    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:01:32.165277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.165249    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:32.165345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.165308    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:32.165345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.165339    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:01:32.303272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.303240    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" event={"ID":"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a","Type":"ContainerStarted","Data":"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"}
Oct 30 19:01:32.303537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.303516    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" event={"ID":"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a","Type":"ContainerStarted","Data":"f87fbeafe04ed3b09601763303287cb91c18da296680bfea7e7366642c571937"}
Oct 30 19:01:32.332491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.332430    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podStartSLOduration=1.332416383 podStartE2EDuration="1.332416383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:32.328800724 +0000 UTC m=+1712.592265674" watchObservedRunningTime="2026-10-30 19:01:32.332416383 +0000 UTC m=+1712.595881330"
Oct 30 19:01:32.824635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:32.824600    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:33.830190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:33.825855    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:33.830190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:33.825925    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:34.493584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:34.477422    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:34.493584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:34.477701    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerName="istio-proxy" containerID="cri-o://96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094" gracePeriod=30
Oct 30 19:01:34.844113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:34.842406    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:34.844113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:34.842484    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:35.047440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:35.047398    5065 patch_prober.go:29] interesting pod/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:01:35.047597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:35.047469    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:01:35.825159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:35.825121    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:35.825365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:35.825193    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:36.825042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:36.824996    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:36.825563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:36.825380    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:37.825498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:37.825454    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:37.826039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:37.825534    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:38.825546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:38.825508    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:01:38.826056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:38.825580    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Oct 30 19:01:39.845419 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:39.845380    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:39.846514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:39.846488    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:39.847529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:39.847507    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:01:40.026236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.026212    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:40.120371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120334    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.120631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.120631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120546    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.120631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120584    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.120631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120621    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.121019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120682    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.121019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120722    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.121019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120788    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.121019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.120932    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\") pod \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\" (UID: \"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5\") "
Oct 30 19:01:40.121317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.121210    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:40.121533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.121502    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:40.121713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.121694    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:40.122153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.122022    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.122153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.122049    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.122153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.122063    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.122365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.122282    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:40.122527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.122485    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:40.125054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.125030    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:40.127428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.127403    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "kube-api-access-fd4c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:40.128800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.128725    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:40.131179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.131146    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy" pod "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" (UID: "aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:40.223967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.223935    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.223967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.223965    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.224179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.223981    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.224179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.223990    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.224179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.223999    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4c7\" (UniqueName: \"kubernetes.io/projected/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-kube-api-access-fd4c7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.224179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.224009    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:40.720784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.720748    5065 generic.go:356] "Generic (PLEG): container finished" podID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerID="96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094" exitCode=0
Oct 30 19:01:40.721119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.720806    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"
Oct 30 19:01:40.721278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.720828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" event={"ID":"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5","Type":"ContainerDied","Data":"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"}
Oct 30 19:01:40.721278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.721245    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph" event={"ID":"aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5","Type":"ContainerDied","Data":"6a9c6c27ddaa478e47ace1650878d690b7cba97dc473be0b5f9b3c4a248e7eea"}
Oct 30 19:01:40.721424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.721286    5065 scope.go:122] "RemoveContainer" containerID="96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"
Oct 30 19:01:40.736129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.736054    5065 scope.go:122] "RemoveContainer" containerID="96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"
Oct 30 19:01:40.736470 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:01:40.736443    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094\": container with ID starting with 96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094 not found: ID does not exist" containerID="96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"
Oct 30 19:01:40.736562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.736484    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094"} err="failed to get container status \"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094\": rpc error: code = NotFound desc = could not find container \"96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094\": container with ID starting with 96ec3d425b34db389db1894b5ad61811138e927111668edda7974a3d45710094 not found: ID does not exist"
Oct 30 19:01:40.756726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.756705    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:40.761977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:40.761956    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-aec752-openshift-default-587f54fb87-qs6ph"]
Oct 30 19:01:41.893691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:41.893660    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" path="/var/lib/kubelet/pods/aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5/volumes"
Oct 30 19:01:43.032192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:43.032160    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"]
Oct 30 19:01:43.037130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:43.037100    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a001c07-hz2bm"]
Oct 30 19:01:43.903109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:43.899201    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68db44fb-7ce3-4b51-a6f1-14ec581f742e" path="/var/lib/kubelet/pods/68db44fb-7ce3-4b51-a6f1-14ec581f742e/volumes"
Oct 30 19:01:45.030976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:45.030936    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"]
Oct 30 19:01:45.035757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:45.035736    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc181d1-cb74h"]
Oct 30 19:01:45.955117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:45.955066    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303b8be0-8b94-4f04-baa3-15f80d7e8739" path="/var/lib/kubelet/pods/303b8be0-8b94-4f04-baa3-15f80d7e8739/volumes"
Oct 30 19:01:53.058475 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:01:53.058438    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:01:58.032380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:58.032340    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"]
Oct 30 19:01:58.036990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:58.036963    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416e8ddc-xxmvj"]
Oct 30 19:01:59.893260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:01:59.893222    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e41470b-5581-4512-a2ca-8cb33069953f" path="/var/lib/kubelet/pods/9e41470b-5581-4512-a2ca-8cb33069953f/volumes"
Oct 30 19:02:08.031810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:08.031773    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"]
Oct 30 19:02:08.036646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:08.036618    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19943312-9jhkf"]
Oct 30 19:02:09.894771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:09.894734    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2e0f72-435d-4667-a71d-ef0c4b93ce4d" path="/var/lib/kubelet/pods/ff2e0f72-435d-4667-a71d-ef0c4b93ce4d/volumes"
Oct 30 19:02:10.517311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.517234    5065 scope.go:122] "RemoveContainer" containerID="adc251a81bc890b62a933b7051fcb424404a49b6548a2217705c7df46748259b"
Oct 30 19:02:10.529622 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.529597    5065 scope.go:122] "RemoveContainer" containerID="6986ec3bdda1dc136920fe69fa8db8a581803f2b376209fe458c1369de624eaf"
Oct 30 19:02:10.543346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.543321    5065 scope.go:122] "RemoveContainer" containerID="3bef2f301786d79e27392639923977292aa6aa15669a3897d9360a03877962f2"
Oct 30 19:02:10.554279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.554261    5065 scope.go:122] "RemoveContainer" containerID="6bc635941028ef8380da24642f81bd8f3f72413e1b99d3c61adb262e9781ee44"
Oct 30 19:02:10.564460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.564444    5065 scope.go:122] "RemoveContainer" containerID="8b65840d9a06a899d51c84dd9040ae4f7bbc79b4e6d5921830e9fc911cbd6872"
Oct 30 19:02:10.575582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.575563    5065 scope.go:122] "RemoveContainer" containerID="5ff658a6819810df2483686c51b263760fe618ad20b4a52502c435713fe40fca"
Oct 30 19:02:10.586223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:10.586203    5065 scope.go:122] "RemoveContainer" containerID="c29579dc1f9291179d8d8670ca4bd421f5e4c2bd19e61a5a8b093df00873eb9c"
Oct 30 19:02:11.112421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:11.112149    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"]
Oct 30 19:02:11.117861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:11.117831    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a1caf0-827ll"]
Oct 30 19:02:11.892776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:11.892749    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5ef851-8a3b-46a4-a3fa-96b30f03e09a" path="/var/lib/kubelet/pods/9f5ef851-8a3b-46a4-a3fa-96b30f03e09a/volumes"
Oct 30 19:02:23.060457 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:02:23.060420    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:02:25.375138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.373860    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:02:25.375138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.374684    5065 cpu_manager.go:414] "containerMap: removing container" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerName="istio-proxy"
Oct 30 19:02:25.375138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.374698    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerName="istio-proxy"
Oct 30 19:02:25.375138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.374806    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff4d114-bb4c-4f1b-85c1-9fcddc6af4b5" containerName="istio-proxy"
Oct 30 19:02:25.378745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.378724    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.393717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.393678    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e1e1ff-openshift-default-dockercfg-l8zv6\""
Oct 30 19:02:25.404540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.404511    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:02:25.409059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.408868    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.408982    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409151    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409203    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409233    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409297    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409324    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h85b\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409361    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.409469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.409404    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566461    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566539    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566654    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566701    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566731    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566798    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566826    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h85b\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566861    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.566904    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.581107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.567561    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.641172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.616516    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.641172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.616829    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.641172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.626395    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.641172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.628297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.680185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.673606    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.739177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.687745    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.739177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.714887    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h85b\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:25.772688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:25.772656    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\") pod \"e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") " pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:26.002457 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:26.002372    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:26.314623 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:26.314588    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:02:26.316728 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:02:26.316699    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c55551_f528_4226_b624_6a9f8409b0f6.slice/crio-633f806644c8131b7c26d018e494953bb425db7e14ee279b5b3ede76f1a47cf9 WatchSource:0}: Error finding container 633f806644c8131b7c26d018e494953bb425db7e14ee279b5b3ede76f1a47cf9: Status 404 returned error can't find the container with id 633f806644c8131b7c26d018e494953bb425db7e14ee279b5b3ede76f1a47cf9
Oct 30 19:02:26.319776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:26.319741    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:02:26.319876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:26.319821    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:02:26.319876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:26.319862    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:02:27.028106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.028049    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"]
Oct 30 19:02:27.032908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.032881    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f087832e-cb2n2"]
Oct 30 19:02:27.327590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.327507    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerStarted","Data":"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"}
Oct 30 19:02:27.327590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.327574    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerStarted","Data":"633f806644c8131b7c26d018e494953bb425db7e14ee279b5b3ede76f1a47cf9"}
Oct 30 19:02:27.360849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.360803    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" podStartSLOduration=2.360786813 podStartE2EDuration="2.360786813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:27.356323665 +0000 UTC m=+1767.619788636" watchObservedRunningTime="2026-10-30 19:02:27.360786813 +0000 UTC m=+1767.624251809"
Oct 30 19:02:27.961981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:27.961948    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a1a7ce-d87e-429a-abe3-d857118f5803" path="/var/lib/kubelet/pods/81a1a7ce-d87e-429a-abe3-d857118f5803/volumes"
Oct 30 19:02:28.003132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:28.003065    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:28.010686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:28.010663    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:28.336724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:28.336690    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:28.337848 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:28.337830    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:02:32.027918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:32.027890    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"]
Oct 30 19:02:32.032376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:32.032354    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a6e513-9tpp7"]
Oct 30 19:02:33.895497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:33.895451    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33957d35-0e51-4f51-a8c2-c8eb8c070f10" path="/var/lib/kubelet/pods/33957d35-0e51-4f51-a8c2-c8eb8c070f10/volumes"
Oct 30 19:02:39.050773 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:39.050739    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"]
Oct 30 19:02:39.054693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:39.054666    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8545393-jfbtd"]
Oct 30 19:02:39.892572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:39.892539    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522ff37a-3eb0-4f56-8800-17e8fa27911a" path="/var/lib/kubelet/pods/522ff37a-3eb0-4f56-8800-17e8fa27911a/volumes"
Oct 30 19:02:46.034792 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:46.034756    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"]
Oct 30 19:02:46.040053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:46.040025    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3b2ca9-52jnw"]
Oct 30 19:02:47.913703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:47.913661    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c68a2f-7584-44ef-80bc-d9c461c71905" path="/var/lib/kubelet/pods/82c68a2f-7584-44ef-80bc-d9c461c71905/volumes"
Oct 30 19:02:53.123406 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:02:53.123357    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:02:54.046840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:54.046809    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"]
Oct 30 19:02:54.051257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:54.051232    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b05d02a-dffxl"]
Oct 30 19:02:55.900045 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:02:55.900012    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804eaaa1-5701-471b-a871-2f59fb19a692" path="/var/lib/kubelet/pods/804eaaa1-5701-471b-a871-2f59fb19a692/volumes"
Oct 30 19:03:02.160450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.160417    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:03:02.171613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.171580    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.190542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.190415    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b0e009-openshift-default-dockercfg-jdddz\""
Oct 30 19:03:02.194158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.194136    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355491    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355593    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gxh\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355695    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355813    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355912    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.355963    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.356203    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.356246    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.356306    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.462545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.462459    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.464168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.464064    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.464457 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.464439    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.464611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.464594    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.464745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.464730    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.464936 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.464918    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.465107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.465071    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gxh\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.475033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.474882    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.475033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.463506    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.476486 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.476026    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.476486 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.476249    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.476980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.476943    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.477549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.477428    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.477778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.477760    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.523175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.521631    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.523175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.522136    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.525254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.525188    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.588010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.582600    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gxh\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\") pod \"e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") " pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:02.801568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:02.801537    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:03.192206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.192175    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:03:03.194335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.194301    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:03.194533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.194516    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:03.194657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.194644    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:03.480718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.480623    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:03:03.490636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.490608    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:03:03.543493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.543462    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:03:03.552048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.552028    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:03:03.903079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.903039    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerStarted","Data":"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"}
Oct 30 19:03:03.903336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.903119    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerStarted","Data":"5326f1756386935af1202efeaae76a5dbea3d3dd3a9730749ef499269de9f673"}
Oct 30 19:03:03.997232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:03.995276    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podStartSLOduration=1.995256916 podStartE2EDuration="1.995256916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:03.990043136 +0000 UTC m=+1804.253508083" watchObservedRunningTime="2026-10-30 19:03:03.995256916 +0000 UTC m=+1804.258721867"
Oct 30 19:03:04.183612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.181071    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"]
Oct 30 19:03:04.221178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.220265    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cf5631-cf5wg"]
Oct 30 19:03:04.802482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.802449    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:04.809010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.808981    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:04.919225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.919172    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:04.921183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:04.921159    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:05.940938 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:05.940907    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077501db-9736-421f-8262-b43ca22469cc" path="/var/lib/kubelet/pods/077501db-9736-421f-8262-b43ca22469cc/volumes"
Oct 30 19:03:06.046551 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:06.046516    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"]
Oct 30 19:03:06.052971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:06.052944    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e8883a-bfvfb"]
Oct 30 19:03:07.915942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:07.915441    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d7b64c-f2c9-480e-9471-35a7c816453a" path="/var/lib/kubelet/pods/59d7b64c-f2c9-480e-9471-35a7c816453a/volumes"
Oct 30 19:03:10.676101 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.676061    5065 scope.go:122] "RemoveContainer" containerID="b9edf8fe4e75e5289e5293927c00548a8171d026cbd3b132d9519bfaf6222ae4"
Oct 30 19:03:10.691185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.691161    5065 scope.go:122] "RemoveContainer" containerID="f5c5d0e7b66ac66262f99c727b11af3a949f13258909d4bef0206b83c5c470a2"
Oct 30 19:03:10.704450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.704428    5065 scope.go:122] "RemoveContainer" containerID="f513083fbf3301a0864de26cbe7db7dc9640e4ed2841ba3d197a37b9ffcf1a99"
Oct 30 19:03:10.717897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.717871    5065 scope.go:122] "RemoveContainer" containerID="31fa70612de9dff7cd156f5009b5889f806796987df23b09f721f8598162b8e8"
Oct 30 19:03:10.730148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.730125    5065 scope.go:122] "RemoveContainer" containerID="dd8f82f35fdffd6074ef23723b1c6c289a163b3659194d156a8f0313ced1b31d"
Oct 30 19:03:10.740197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.740177    5065 scope.go:122] "RemoveContainer" containerID="5230794bb7fac8e3e1c868f152193e127e32e475bc180fe2df4caa4fa55345c2"
Oct 30 19:03:10.750648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.750630    5065 scope.go:122] "RemoveContainer" containerID="7493f49d7e94509da72fcefd5f565179e60a1e607c0c17f6333e695270d191d0"
Oct 30 19:03:10.764270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.764251    5065 scope.go:122] "RemoveContainer" containerID="cb61070e4af50a11a41e8887b9eaf20f8ac0430d1fde484a2be8bf199efab904"
Oct 30 19:03:10.774401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.774380    5065 scope.go:122] "RemoveContainer" containerID="f2fc65445728d5595adb8e0e05c9942aa92e7e585b593db896c128ae9738a028"
Oct 30 19:03:10.785569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:10.785553    5065 scope.go:122] "RemoveContainer" containerID="b6688ddb0aac3cab6120a0bb694769ebf306755a9a83ad2f6f4727a8cae83fc5"
Oct 30 19:03:17.076465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:17.076423    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"]
Oct 30 19:03:17.085926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:17.085897    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c7df416-qg8wv"]
Oct 30 19:03:17.895448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:17.895413    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28eb5945-f438-4b9f-9f18-4fef919850bf" path="/var/lib/kubelet/pods/28eb5945-f438-4b9f-9f18-4fef919850bf/volumes"
Oct 30 19:03:23.061502 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:03:23.061465    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:03:24.131656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:24.131617    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"]
Oct 30 19:03:24.136597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:24.136569    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af0ab66-l5c9v"]
Oct 30 19:03:25.893538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:25.893500    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cbdc2b-0841-4ac7-9c83-55284b4fec35" path="/var/lib/kubelet/pods/19cbdc2b-0841-4ac7-9c83-55284b4fec35/volumes"
Oct 30 19:03:37.097114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:37.096661    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"]
Oct 30 19:03:37.127111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:37.123811    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab88ccf-799kt"]
Oct 30 19:03:38.004109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:38.003356    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e189df7-1e47-487c-8b2f-934ecb60b235" path="/var/lib/kubelet/pods/0e189df7-1e47-487c-8b2f-934ecb60b235/volumes"
Oct 30 19:03:42.957752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:42.957673    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"]
Oct 30 19:03:42.963958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:42.963938    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:42.980243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:42.980219    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:03:42.980497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:42.980480    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:43.003135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:42.997912    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"]
Oct 30 19:03:43.048930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.048896    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4r9v\" (UniqueName: \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.049131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.049053    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.049373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.049356    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.150856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.150818    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r9v\" (UniqueName: \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.151055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.150937    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.151241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.151224    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.151826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.151809    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.153416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.153393    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.172595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.172569    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4r9v\" (UniqueName: \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\") pod \"maas-api-revoke-sub-6942ce87-qk2jv\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.305116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.301161    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:43.662937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.662899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"]
Oct 30 19:03:43.666032 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:03:43.666005    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fdb365_7e71_4f28_8265_106b10237ac7.slice/crio-ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762 WatchSource:0}: Error finding container ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762: Status 404 returned error can't find the container with id ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762
Oct 30 19:03:43.772233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:43.772200    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv" event={"ID":"82fdb365-7e71-4f28-8265-106b10237ac7","Type":"ContainerStarted","Data":"ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762"}
Oct 30 19:03:44.783300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:44.783267    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv" event={"ID":"82fdb365-7e71-4f28-8265-106b10237ac7","Type":"ContainerStarted","Data":"9455acaf755b88ee3ad6d80cbe7aecdf33ac0442dcb91e882568a07199340a19"}
Oct 30 19:03:44.815271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:44.815215    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv" podStartSLOduration=2.8151960799999998 podStartE2EDuration="2.81519608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.810787868 +0000 UTC m=+1845.074252847" watchObservedRunningTime="2026-10-30 19:03:44.81519608 +0000 UTC m=+1845.078661031"
Oct 30 19:03:45.554064 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.553172    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:03:45.573325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.573297    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:03:45.573560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.573403    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.582447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.582421    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a717c7-openshift-default-dockercfg-v8bzd\""
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.721627    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.721774    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.721842    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.721884    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.721972    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.722075    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.722398    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.722431    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.728159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.722522    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktlg\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.875579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.875430    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktlg\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.875944    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876181    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876276    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876425    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876525    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876572    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.876653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.876611    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.878287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.878234    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.890124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.890069    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.904401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.904192    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.905156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.904823    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.905156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.905118    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.915243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.909602    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.915243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.913976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.926047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.925473    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktlg\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.935103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.931250    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\") pod \"e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") " pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:45.935103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.931502    5065 generic.go:356] "Generic (PLEG): container finished" podID="82fdb365-7e71-4f28-8265-106b10237ac7" containerID="9455acaf755b88ee3ad6d80cbe7aecdf33ac0442dcb91e882568a07199340a19" exitCode=0
Oct 30 19:03:45.935103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:45.931879    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv" event={"ID":"82fdb365-7e71-4f28-8265-106b10237ac7","Type":"ContainerDied","Data":"9455acaf755b88ee3ad6d80cbe7aecdf33ac0442dcb91e882568a07199340a19"}
Oct 30 19:03:46.193746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.193666    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:46.501844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.501814    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:03:46.504872 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:03:46.504838    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda047c669_648f_4d9b_86ef_866441a1330b.slice/crio-14b44b96579454965a9fd4050e11c7fb4e52dcdbde0ada0b00d4cef206b06ba6 WatchSource:0}: Error finding container 14b44b96579454965a9fd4050e11c7fb4e52dcdbde0ada0b00d4cef206b06ba6: Status 404 returned error can't find the container with id 14b44b96579454965a9fd4050e11c7fb4e52dcdbde0ada0b00d4cef206b06ba6
Oct 30 19:03:46.508524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.508479    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:46.508649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.508574    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:46.508649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.508615    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:46.947552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.945293    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerStarted","Data":"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"}
Oct 30 19:03:46.947552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.945363    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerStarted","Data":"14b44b96579454965a9fd4050e11c7fb4e52dcdbde0ada0b00d4cef206b06ba6"}
Oct 30 19:03:46.980871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:46.979497    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podStartSLOduration=1.979477477 podStartE2EDuration="1.979477477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:46.975126936 +0000 UTC m=+1847.238591894" watchObservedRunningTime="2026-10-30 19:03:46.979477477 +0000 UTC m=+1847.242942425"
Oct 30 19:03:47.205507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.197721    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:47.205507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.197903    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:03:47.205507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.197950    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:03:47.264896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.264866    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:47.443983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.443952    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\" (UniqueName: \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\") pod \"82fdb365-7e71-4f28-8265-106b10237ac7\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") "
Oct 30 19:03:47.444212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.443995    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\") pod \"82fdb365-7e71-4f28-8265-106b10237ac7\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") "
Oct 30 19:03:47.444212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.444065    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\") pod \"82fdb365-7e71-4f28-8265-106b10237ac7\" (UID: \"82fdb365-7e71-4f28-8265-106b10237ac7\") "
Oct 30 19:03:47.444476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.444446    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca" pod "82fdb365-7e71-4f28-8265-106b10237ac7" (UID: "82fdb365-7e71-4f28-8265-106b10237ac7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:47.444943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.444924    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:47.446642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.446615    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca" pod "82fdb365-7e71-4f28-8265-106b10237ac7" (UID: "82fdb365-7e71-4f28-8265-106b10237ac7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:47.446995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.446973    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v" pod "82fdb365-7e71-4f28-8265-106b10237ac7" (UID: "82fdb365-7e71-4f28-8265-106b10237ac7"). InnerVolumeSpecName "kube-api-access-r4r9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:47.546877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.546834    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4r9v\" (UniqueName: \"kubernetes.io/projected/82fdb365-7e71-4f28-8265-106b10237ac7-kube-api-access-r4r9v\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:47.546877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.546879    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fdb365-7e71-4f28-8265-106b10237ac7-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:47.957175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.957142    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"
Oct 30 19:03:47.957175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.957145    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv" event={"ID":"82fdb365-7e71-4f28-8265-106b10237ac7","Type":"ContainerDied","Data":"ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762"}
Oct 30 19:03:47.957713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:47.957203    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3c5fc0967cb256d87d4e9700abb6f0c74200a8fe6620b88e8f329790ccd762"
Oct 30 19:03:48.200785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:48.200711    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:48.979036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:48.979007    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:48.982815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:48.982786    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:03:49.975788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:49.974681    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": read tcp 10.132.0.2:42188->10.132.1.8:15021: read: connection reset by peer" start-of-body=
Oct 30 19:03:49.975788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:49.974744    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": read tcp 10.132.0.2:42188->10.132.1.8:15021: read: connection reset by peer"
Oct 30 19:03:50.819840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.819796    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:03:50.820648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.820630    5065 cpu_manager.go:414] "containerMap: removing container" podUID="82fdb365-7e71-4f28-8265-106b10237ac7" containerName="revoke-keys"
Oct 30 19:03:50.820648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.820647    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fdb365-7e71-4f28-8265-106b10237ac7" containerName="revoke-keys"
Oct 30 19:03:50.820794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.820779    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fdb365-7e71-4f28-8265-106b10237ac7" containerName="revoke-keys"
Oct 30 19:03:50.824536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.824507    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:50.833961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.833933    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a717c7\""
Oct 30 19:03:50.837370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.837347    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:03:50.842286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.842268    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:50.844145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.844121    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:03:50.853136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.849925    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a717c7-dockercfg-2zdpq\""
Oct 30 19:03:50.864439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.864414    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:03:50.923126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.923050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:50.923445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.923423    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnlk\" (UniqueName: \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:50.923654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.923639    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fcs\" (UniqueName: \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:50.923834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:50.923820    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.035107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.031552    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fcs\" (UniqueName: \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.035107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.031680    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.039339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.039305    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:51.039708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.039669    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnlk\" (UniqueName: \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:51.047236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.040314    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.047236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.041047    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:51.084239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.076190    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnlk\" (UniqueName: \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\") pod \"payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:51.145344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.145054    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:51.146528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.146499    5065 generic.go:356] "Generic (PLEG): container finished" podID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerID="be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e" exitCode=0
Oct 30 19:03:51.162119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.147673    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerDied","Data":"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"}
Oct 30 19:03:51.185771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.157524    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fcs\" (UniqueName: \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\") pod \"payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.185771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.166396    5065 scope.go:122] "RemoveContainer" containerID="be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"
Oct 30 19:03:51.241485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.241440    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:51.241776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.241753    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:51.248346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.248309    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:03:51.470473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.458687    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:51.767946 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:03:51.767915    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecf4571_f593_450f_b159_8c256405fdb0.slice/crio-b05004e020acfdfedc62ef41c319db25557893e7e290b49515f405c5c381448e WatchSource:0}: Error finding container b05004e020acfdfedc62ef41c319db25557893e7e290b49515f405c5c381448e: Status 404 returned error can't find the container with id b05004e020acfdfedc62ef41c319db25557893e7e290b49515f405c5c381448e
Oct 30 19:03:51.769581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.769556    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:03:51.794288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:51.794265    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:03:51.802201 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:03:51.802168    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf8a994_50f0_4224_afa7_1715b4527ad0.slice/crio-21d26de98efc0c06109b49cf849845eb4f66751723c089fcef22279ccaa32404 WatchSource:0}: Error finding container 21d26de98efc0c06109b49cf849845eb4f66751723c089fcef22279ccaa32404: Status 404 returned error can't find the container with id 21d26de98efc0c06109b49cf849845eb4f66751723c089fcef22279ccaa32404
Oct 30 19:03:52.164548 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.163192    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerStarted","Data":"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"}
Oct 30 19:03:52.170782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.166880    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" event={"ID":"6ecf4571-f593-450f-b159-8c256405fdb0","Type":"ContainerStarted","Data":"38e2a8da6e7a17e21a58f59cb9ce596381762eae9675e25428a75a4f3e3e3913"}
Oct 30 19:03:52.170782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.166939    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" event={"ID":"6ecf4571-f593-450f-b159-8c256405fdb0","Type":"ContainerStarted","Data":"b05004e020acfdfedc62ef41c319db25557893e7e290b49515f405c5c381448e"}
Oct 30 19:03:52.170782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.167837    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:03:52.172676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.172633    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" event={"ID":"3cf8a994-50f0-4224-afa7-1715b4527ad0","Type":"ContainerStarted","Data":"cf3a042c405d28b1f9a4bd5f1119b941c3e5606a18d3e9c6a95b47595a4ce04b"}
Oct 30 19:03:52.172835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.172687    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" event={"ID":"3cf8a994-50f0-4224-afa7-1715b4527ad0","Type":"ContainerStarted","Data":"21d26de98efc0c06109b49cf849845eb4f66751723c089fcef22279ccaa32404"}
Oct 30 19:03:52.173424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.173403    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:03:52.231988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.231913    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" podStartSLOduration=2.23189958 podStartE2EDuration="2.23189958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.225186721 +0000 UTC m=+1852.488651688" watchObservedRunningTime="2026-10-30 19:03:52.23189958 +0000 UTC m=+1852.495364528"
Oct 30 19:03:52.250588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.250543    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" podStartSLOduration=2.250529521 podStartE2EDuration="2.250529521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.248116343 +0000 UTC m=+1852.511581294" watchObservedRunningTime="2026-10-30 19:03:52.250529521 +0000 UTC m=+1852.513994469"
Oct 30 19:03:52.804650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.804558    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:52.804971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.804946    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:03:52.805154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:52.805130    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 19:03:53.149761 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:03:53.149403    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:03:53.325488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.325426    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"]
Oct 30 19:03:53.361940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.360271    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"]
Oct 30 19:03:53.361940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.360378    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.367643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.367613    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:53.368743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.368725    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:03:53.470181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.470080    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.470335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.470237    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q56z\" (UniqueName: \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.470471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.470455    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.571933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.571901    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.572154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.571980    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.572154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.572067    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q56z\" (UniqueName: \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.572552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.572530    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.574540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.574519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.600844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.600776    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q56z\" (UniqueName: \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\") pod \"maas-api-revoke-sub-07eff279-pmrdw\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.710322 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.710288    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:53.833110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.827107    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:03:53.833110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:53.827159    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 19:03:54.072643 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:03:54.072363    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef7aa3a_3dc0_4b61_9076_9e680758c292.slice/crio-94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce WatchSource:0}: Error finding container 94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce: Status 404 returned error can't find the container with id 94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce
Oct 30 19:03:54.074720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:54.074693    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"]
Oct 30 19:03:54.221943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:54.221909    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw" event={"ID":"5ef7aa3a-3dc0-4b61-9076-9e680758c292","Type":"ContainerStarted","Data":"94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce"}
Oct 30 19:03:54.802161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:54.802126    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:03:54.802718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:54.802682    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 19:03:55.232469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:55.232417    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw" event={"ID":"5ef7aa3a-3dc0-4b61-9076-9e680758c292","Type":"ContainerStarted","Data":"96fd5fa55b7d24a178d840204a00d9d3417361e3e8b5a170344aa994053007a6"}
Oct 30 19:03:55.259181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:55.259119    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw" podStartSLOduration=2.259101673 podStartE2EDuration="2.259101673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:55.252631634 +0000 UTC m=+1855.516096582" watchObservedRunningTime="2026-10-30 19:03:55.259101673 +0000 UTC m=+1855.522566614"
Oct 30 19:03:55.804108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:55.803577    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:03:55.804108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:55.803654    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 19:03:56.243291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:56.243210    5065 generic.go:356] "Generic (PLEG): container finished" podID="5ef7aa3a-3dc0-4b61-9076-9e680758c292" containerID="96fd5fa55b7d24a178d840204a00d9d3417361e3e8b5a170344aa994053007a6" exitCode=0
Oct 30 19:03:56.243450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:56.243306    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw" event={"ID":"5ef7aa3a-3dc0-4b61-9076-9e680758c292","Type":"ContainerDied","Data":"96fd5fa55b7d24a178d840204a00d9d3417361e3e8b5a170344aa994053007a6"}
Oct 30 19:03:56.804471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:56.804441    5065 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:03:56.804953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:56.804552    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 19:03:57.609174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.609131    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:57.670789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.670756    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\") pod \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") "
Oct 30 19:03:57.670993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.670958    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\" (UniqueName: \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\") pod \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") "
Oct 30 19:03:57.671148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.671129    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\") pod \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\" (UID: \"5ef7aa3a-3dc0-4b61-9076-9e680758c292\") "
Oct 30 19:03:57.671534 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.671500    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca" pod "5ef7aa3a-3dc0-4b61-9076-9e680758c292" (UID: "5ef7aa3a-3dc0-4b61-9076-9e680758c292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:57.671974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.671950    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:57.674111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.673997    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca" pod "5ef7aa3a-3dc0-4b61-9076-9e680758c292" (UID: "5ef7aa3a-3dc0-4b61-9076-9e680758c292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:57.674648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.674622    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z" pod "5ef7aa3a-3dc0-4b61-9076-9e680758c292" (UID: "5ef7aa3a-3dc0-4b61-9076-9e680758c292"). InnerVolumeSpecName "kube-api-access-7q56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:57.773438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.773403    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef7aa3a-3dc0-4b61-9076-9e680758c292-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:57.773438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.773438    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q56z\" (UniqueName: \"kubernetes.io/projected/5ef7aa3a-3dc0-4b61-9076-9e680758c292-kube-api-access-7q56z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:57.836265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.836164    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:57.836772 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.836721    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:57.837997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:57.837974    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:03:58.033262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:58.033234    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"]
Oct 30 19:03:58.061181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:58.040816    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cdbb56-c4g9c"]
Oct 30 19:03:58.314034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:58.314000    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"
Oct 30 19:03:58.314034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:58.314024    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw" event={"ID":"5ef7aa3a-3dc0-4b61-9076-9e680758c292","Type":"ContainerDied","Data":"94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce"}
Oct 30 19:03:58.314336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:58.314149    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94caaad89b018dec8f1d3d525d6e10a612af12f9c2f1d7ea83f48593552980ce"
Oct 30 19:03:59.897169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:03:59.897134    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d583a3-bf7d-4256-ad71-1f8bad5d75f0" path="/var/lib/kubelet/pods/21d583a3-bf7d-4256-ad71-1f8bad5d75f0/volumes"
Oct 30 19:04:04.240114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:04.237007    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:04:04.240114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:04.237123    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:04:07.034326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:07.034294    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"]
Oct 30 19:04:07.039546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:07.039523    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107eded0-rgz9z"]
Oct 30 19:04:07.894542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:07.894511    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f" path="/var/lib/kubelet/pods/786fa9c7-cd56-4216-a5ad-51b8b3a6fa7f/volumes"
Oct 30 19:04:10.896843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.896808    5065 scope.go:122] "RemoveContainer" containerID="5c0b7f8591cdf4dff23fc1b7d8373d20c58906411116f6e9f56e462f58ced2a9"
Oct 30 19:04:10.910132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.910073    5065 scope.go:122] "RemoveContainer" containerID="cead6db4143b8aa0d5c8b28ddbe60229a5216086550955cc6ee756a62820c2a4"
Oct 30 19:04:10.921129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.921106    5065 scope.go:122] "RemoveContainer" containerID="13f69476f08b6159029bacedf96081b96210872da5ac205dca13e454d5f0a75c"
Oct 30 19:04:10.935157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.935132    5065 scope.go:122] "RemoveContainer" containerID="442a524ff4cbdf746f38ac778460cf242a98f7533e50c7df5b69b1f070c5d74a"
Oct 30 19:04:10.947151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.947134    5065 scope.go:122] "RemoveContainer" containerID="d5ef7e66c787e8a0480ee48ed558fe369739597a84b335f645d288e4c9ac95c1"
Oct 30 19:04:10.956990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.956974    5065 scope.go:122] "RemoveContainer" containerID="9eab4ba578651283f160ad5b6332c93ae6c26254ae881bbbdeb24ce76979aff4"
Oct 30 19:04:10.966387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.966367    5065 scope.go:122] "RemoveContainer" containerID="04f4d68954e4cbea83d2d4730bf0f25cee1aabd3a36a3372f1a9bcbaec98950f"
Oct 30 19:04:10.975656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.975638    5065 scope.go:122] "RemoveContainer" containerID="fbea7547abf46f3ac944d5be0bacaf743d2c39d8fb5335561ff66c7c3f3137e8"
Oct 30 19:04:10.988361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:10.988342    5065 scope.go:122] "RemoveContainer" containerID="36a863e5e469c2575f19d7900436bf03f4e01f36c99cdc889d05792d9d434c71"
Oct 30 19:04:17.209134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.209102    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"]
Oct 30 19:04:17.209897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.209883    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5ef7aa3a-3dc0-4b61-9076-9e680758c292" containerName="revoke-keys"
Oct 30 19:04:17.209962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.209899    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef7aa3a-3dc0-4b61-9076-9e680758c292" containerName="revoke-keys"
Oct 30 19:04:17.210036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.210027    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef7aa3a-3dc0-4b61-9076-9e680758c292" containerName="revoke-keys"
Oct 30 19:04:17.215847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.215832    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.225945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.225883    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:17.226155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.226139    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:04:17.243031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.243005    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"]
Oct 30 19:04:17.303776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.303743    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.303974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.303805    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.304071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.304048    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgnp\" (UniqueName: \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.326012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.325982    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"]
Oct 30 19:04:17.330776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.330760    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.339170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.339142    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"]
Oct 30 19:04:17.405151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405116    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txjt\" (UniqueName: \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.405313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405165    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgnp\" (UniqueName: \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.405377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405348    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.405448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405393    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.405448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405415    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.405558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.405485    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.406021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.406003    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.407882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.407861    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.423824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.423800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgnp\" (UniqueName: \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\") pod \"maas-api-revoke-sub-35f11969-xzdzp\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.506816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.506749    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txjt\" (UniqueName: \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.507039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.507020    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.507131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.507074    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.507688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.507671    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.509462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.509436    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.521685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.521661    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txjt\" (UniqueName: \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\") pod \"maas-api-revoke-sub-b10cfdb0-tzk46\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.532570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.532547    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:17.661118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.654205    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:17.830639 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:17.830587    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f30ffbd_d331_4948_ac30_24c1392a6041.slice/crio-423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271 WatchSource:0}: Error finding container 423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271: Status 404 returned error can't find the container with id 423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271
Oct 30 19:04:17.831075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.831031    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"]
Oct 30 19:04:17.909225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:17.909151    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp" event={"ID":"0f30ffbd-d331-4948-ac30-24c1392a6041","Type":"ContainerStarted","Data":"423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271"}
Oct 30 19:04:18.027466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.027438    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"]
Oct 30 19:04:18.033936 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:18.033901    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc3895a_03e2_4ff7_b77a_016f38d72ca1.slice/crio-d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2 WatchSource:0}: Error finding container d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2: Status 404 returned error can't find the container with id d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2
Oct 30 19:04:18.595614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.595570    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"]
Oct 30 19:04:18.601037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.601012    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.612530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.612494    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"]
Oct 30 19:04:18.657116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.652365    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.657116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.652664    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbm4\" (UniqueName: \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.657116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.652740    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.754147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.754114    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbm4\" (UniqueName: \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.754443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.754422    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.754750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.754730    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.755205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.755179    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.758594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.758573    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.769180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.769161    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbm4\" (UniqueName: \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\") pod \"maas-api-revoke-sub-be2d3ede-pmmvb\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.920852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.920770    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp" event={"ID":"0f30ffbd-d331-4948-ac30-24c1392a6041","Type":"ContainerStarted","Data":"5ceb015669e307514be91a9df55a47768c16005b919eba3ae34867b37114cef5"}
Oct 30 19:04:18.922965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.922934    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46" event={"ID":"ffc3895a-03e2-4ff7-b77a-016f38d72ca1","Type":"ContainerStarted","Data":"541dd6303863c3f6a210389c66d9a8ed9f06087dc1f888ebc8bc7ea3be935364"}
Oct 30 19:04:18.923185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.922984    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46" event={"ID":"ffc3895a-03e2-4ff7-b77a-016f38d72ca1","Type":"ContainerStarted","Data":"d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2"}
Oct 30 19:04:18.925554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.925531    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:18.964609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:18.964548    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp" podStartSLOduration=1.964530892 podStartE2EDuration="1.964530892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:18.963685519 +0000 UTC m=+1879.227150491" watchObservedRunningTime="2026-10-30 19:04:18.964530892 +0000 UTC m=+1879.227995847"
Oct 30 19:04:19.214758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.214670    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46" podStartSLOduration=2.214655125 podStartE2EDuration="2.214655125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:19.023810501 +0000 UTC m=+1879.287275480" watchObservedRunningTime="2026-10-30 19:04:19.214655125 +0000 UTC m=+1879.478120074"
Oct 30 19:04:19.219067 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:19.219041    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f58e5e_0179_4950_9bbc_0d9dc258790c.slice/crio-dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047 WatchSource:0}: Error finding container dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047: Status 404 returned error can't find the container with id dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047
Oct 30 19:04:19.219542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.219521    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"]
Oct 30 19:04:19.442134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.442105    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 19:04:19.442340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.442316    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy" containerID="cri-o://8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af" gracePeriod=30
Oct 30 19:04:19.982168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.981773    5065 generic.go:356] "Generic (PLEG): container finished" podID="ffc3895a-03e2-4ff7-b77a-016f38d72ca1" containerID="541dd6303863c3f6a210389c66d9a8ed9f06087dc1f888ebc8bc7ea3be935364" exitCode=0
Oct 30 19:04:19.982168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.982048    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46" event={"ID":"ffc3895a-03e2-4ff7-b77a-016f38d72ca1","Type":"ContainerDied","Data":"541dd6303863c3f6a210389c66d9a8ed9f06087dc1f888ebc8bc7ea3be935364"}
Oct 30 19:04:19.995253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.995216    5065 generic.go:356] "Generic (PLEG): container finished" podID="0f30ffbd-d331-4948-ac30-24c1392a6041" containerID="5ceb015669e307514be91a9df55a47768c16005b919eba3ae34867b37114cef5" exitCode=0
Oct 30 19:04:19.996379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.996348    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp" event={"ID":"0f30ffbd-d331-4948-ac30-24c1392a6041","Type":"ContainerDied","Data":"5ceb015669e307514be91a9df55a47768c16005b919eba3ae34867b37114cef5"}
Oct 30 19:04:19.999993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:19.999961    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb" event={"ID":"d0f58e5e-0179-4950-9bbc-0d9dc258790c","Type":"ContainerStarted","Data":"bc689fe03482b2ab9dc87ced189dbfb9ca75b86f61d32771e6869e584ba872df"}
Oct 30 19:04:20.000207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:20.000021    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb" event={"ID":"d0f58e5e-0179-4950-9bbc-0d9dc258790c","Type":"ContainerStarted","Data":"dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047"}
Oct 30 19:04:20.060734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:20.060682    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb" podStartSLOduration=2.0606694230000002 podStartE2EDuration="2.060669423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:20.056411321 +0000 UTC m=+1880.319876268" watchObservedRunningTime="2026-10-30 19:04:20.060669423 +0000 UTC m=+1880.324134370"
Oct 30 19:04:21.090114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.089624    5065 generic.go:356] "Generic (PLEG): container finished" podID="d0f58e5e-0179-4950-9bbc-0d9dc258790c" containerID="bc689fe03482b2ab9dc87ced189dbfb9ca75b86f61d32771e6869e584ba872df" exitCode=0
Oct 30 19:04:21.090738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.090500    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb" event={"ID":"d0f58e5e-0179-4950-9bbc-0d9dc258790c","Type":"ContainerDied","Data":"bc689fe03482b2ab9dc87ced189dbfb9ca75b86f61d32771e6869e584ba872df"}
Oct 30 19:04:21.471446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.471395    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:21.554218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.554185    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:21.565335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565311    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\") pod \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") "
Oct 30 19:04:21.565473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565345    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\" (UniqueName: \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\") pod \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") "
Oct 30 19:04:21.565473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565382    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\") pod \"0f30ffbd-d331-4948-ac30-24c1392a6041\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") "
Oct 30 19:04:21.565598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565556    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\" (UniqueName: \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\") pod \"0f30ffbd-d331-4948-ac30-24c1392a6041\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") "
Oct 30 19:04:21.565660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565619    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\") pod \"0f30ffbd-d331-4948-ac30-24c1392a6041\" (UID: \"0f30ffbd-d331-4948-ac30-24c1392a6041\") "
Oct 30 19:04:21.565719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565687    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca" pod "ffc3895a-03e2-4ff7-b77a-016f38d72ca1" (UID: "ffc3895a-03e2-4ff7-b77a-016f38d72ca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:21.565782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565762    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\") pod \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\" (UID: \"ffc3895a-03e2-4ff7-b77a-016f38d72ca1\") "
Oct 30 19:04:21.566013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.565968    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca" pod "0f30ffbd-d331-4948-ac30-24c1392a6041" (UID: "0f30ffbd-d331-4948-ac30-24c1392a6041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:21.566779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.566755    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.566871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.566787    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.568243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.568220    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca" pod "ffc3895a-03e2-4ff7-b77a-016f38d72ca1" (UID: "ffc3895a-03e2-4ff7-b77a-016f38d72ca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:21.568243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.568230    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca" pod "0f30ffbd-d331-4948-ac30-24c1392a6041" (UID: "0f30ffbd-d331-4948-ac30-24c1392a6041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:21.569423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.569400    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt" pod "ffc3895a-03e2-4ff7-b77a-016f38d72ca1" (UID: "ffc3895a-03e2-4ff7-b77a-016f38d72ca1"). InnerVolumeSpecName "kube-api-access-6txjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:21.569626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.569604    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp" pod "0f30ffbd-d331-4948-ac30-24c1392a6041" (UID: "0f30ffbd-d331-4948-ac30-24c1392a6041"). InnerVolumeSpecName "kube-api-access-nkgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:21.668549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.668474    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txjt\" (UniqueName: \"kubernetes.io/projected/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-kube-api-access-6txjt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.668549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.668502    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f30ffbd-d331-4948-ac30-24c1392a6041-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.668549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.668512    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgnp\" (UniqueName: \"kubernetes.io/projected/0f30ffbd-d331-4948-ac30-24c1392a6041-kube-api-access-nkgnp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.668549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:21.668521    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3895a-03e2-4ff7-b77a-016f38d72ca1-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:22.148463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.147347    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp" event={"ID":"0f30ffbd-d331-4948-ac30-24c1392a6041","Type":"ContainerDied","Data":"423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271"}
Oct 30 19:04:22.148463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.147444    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423a09c1130672fe6dac1592d047ac208f71cd8dca00d8364e8fb926c52a3271"
Oct 30 19:04:22.148463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.147595    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"
Oct 30 19:04:22.150017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.149834    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"
Oct 30 19:04:22.150017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.149872    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46" event={"ID":"ffc3895a-03e2-4ff7-b77a-016f38d72ca1","Type":"ContainerDied","Data":"d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2"}
Oct 30 19:04:22.150017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.149914    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01066144928288e970528d4ac2a1b67bf8066d4d299de9d4bb395e9f85ec9c2"
Oct 30 19:04:22.298766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.298738    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:22.494915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.494879    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\") pod \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") "
Oct 30 19:04:22.495124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.495054    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\" (UniqueName: \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\") pod \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") "
Oct 30 19:04:22.495210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.495194    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\") pod \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\" (UID: \"d0f58e5e-0179-4950-9bbc-0d9dc258790c\") "
Oct 30 19:04:22.495272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.495228    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca" pod "d0f58e5e-0179-4950-9bbc-0d9dc258790c" (UID: "d0f58e5e-0179-4950-9bbc-0d9dc258790c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:22.495852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.495828    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:22.497355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.497332    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4" pod "d0f58e5e-0179-4950-9bbc-0d9dc258790c" (UID: "d0f58e5e-0179-4950-9bbc-0d9dc258790c"). InnerVolumeSpecName "kube-api-access-mjbm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:22.497605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.497583    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca" pod "d0f58e5e-0179-4950-9bbc-0d9dc258790c" (UID: "d0f58e5e-0179-4950-9bbc-0d9dc258790c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:22.596829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.596802    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbm4\" (UniqueName: \"kubernetes.io/projected/d0f58e5e-0179-4950-9bbc-0d9dc258790c-kube-api-access-mjbm4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:22.596829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:22.596828    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f58e5e-0179-4950-9bbc-0d9dc258790c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:23.063152 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:23.063068    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:04:23.179057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:23.177108    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb" event={"ID":"d0f58e5e-0179-4950-9bbc-0d9dc258790c","Type":"ContainerDied","Data":"dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047"}
Oct 30 19:04:23.179057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:23.177180    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfc986e86d6cd42e5b60435214be1066b6c5d7a150d7b2919d4a915402b40047"
Oct 30 19:04:23.179057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:23.177272    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"
Oct 30 19:04:24.784986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.784952    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 19:04:24.852709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852681    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852772    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852803    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852898    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852924    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852953    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852995    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.852995    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853039    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853072    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\") pod \"b66d2d86-ee6a-46cd-a891-942f083d3f42\" (UID: \"b66d2d86-ee6a-46cd-a891-942f083d3f42\") "
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853453    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853541    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853914    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853930    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66d2d86-ee6a-46cd-a891-942f083d3f42-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.854134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.853939    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.854968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.854221    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.854968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.854310    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.855686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.855659    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "kube-api-access-8rbzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.856048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.856013    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.856212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.856055    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.859120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.857026    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo" pod "b66d2d86-ee6a-46cd-a891-942f083d3f42" (UID: "b66d2d86-ee6a-46cd-a891-942f083d3f42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:24.955531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955501    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rbzr\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-kube-api-access-8rbzr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.955531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955530    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.955744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955540    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.955744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955548    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.955744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955557    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66d2d86-ee6a-46cd-a891-942f083d3f42-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.955744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:24.955565    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66d2d86-ee6a-46cd-a891-942f083d3f42-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.201968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.201928    5065 generic.go:356] "Generic (PLEG): container finished" podID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerID="8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af" exitCode=0
Oct 30 19:04:25.202230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.201975    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" event={"ID":"b66d2d86-ee6a-46cd-a891-942f083d3f42","Type":"ContainerDied","Data":"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"}
Oct 30 19:04:25.202230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.202025    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm" event={"ID":"b66d2d86-ee6a-46cd-a891-942f083d3f42","Type":"ContainerDied","Data":"2025ee49ea6cf541fcd8e54be99b47b7f014a96ad996430de5265107def9f9b6"}
Oct 30 19:04:25.202230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.202061    5065 scope.go:122] "RemoveContainer" containerID="8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"
Oct 30 19:04:25.202472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.202357    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"
Oct 30 19:04:25.232625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.232605    5065 scope.go:122] "RemoveContainer" containerID="8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"
Oct 30 19:04:25.232954 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:25.232935    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af\": container with ID starting with 8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af not found: ID does not exist" containerID="8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"
Oct 30 19:04:25.233024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.232964    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af"} err="failed to get container status \"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af\": rpc error: code = NotFound desc = could not find container \"8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af\": container with ID starting with 8ca0a6b7cf81cd842615f0f8969b0b55e38727599bf32dfd92f5f7b0fa0d06af not found: ID does not exist"
Oct 30 19:04:25.243198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.243175    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 19:04:25.253438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.253406    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-p6qnm"]
Oct 30 19:04:25.907110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:25.900810    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" path="/var/lib/kubelet/pods/b66d2d86-ee6a-46cd-a891-942f083d3f42/volumes"
Oct 30 19:04:26.932007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.931971    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:04:26.933260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933237    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy"
Oct 30 19:04:26.933260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933259    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933290    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d0f58e5e-0179-4950-9bbc-0d9dc258790c" containerName="revoke-keys"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933298    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f58e5e-0179-4950-9bbc-0d9dc258790c" containerName="revoke-keys"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933315    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0f30ffbd-d331-4948-ac30-24c1392a6041" containerName="revoke-keys"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933323    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f30ffbd-d331-4948-ac30-24c1392a6041" containerName="revoke-keys"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933336    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ffc3895a-03e2-4ff7-b77a-016f38d72ca1" containerName="revoke-keys"
Oct 30 19:04:26.933505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933342    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc3895a-03e2-4ff7-b77a-016f38d72ca1" containerName="revoke-keys"
Oct 30 19:04:26.933821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933575    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66d2d86-ee6a-46cd-a891-942f083d3f42" containerName="istio-proxy"
Oct 30 19:04:26.933821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933591    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f58e5e-0179-4950-9bbc-0d9dc258790c" containerName="revoke-keys"
Oct 30 19:04:26.933821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933602    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f30ffbd-d331-4948-ac30-24c1392a6041" containerName="revoke-keys"
Oct 30 19:04:26.933821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.933611    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc3895a-03e2-4ff7-b77a-016f38d72ca1" containerName="revoke-keys"
Oct 30 19:04:26.940043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.940011    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:04:26.940244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.940149    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.953851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.953831    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f7c6e3-openshift-default-dockercfg-9qn52\""
Oct 30 19:04:26.999207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv78g\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999226    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999279    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999321    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999384    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999438    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999515    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999543    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:26.999637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:26.999606    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.101635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.101596    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.101837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.101648    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.101837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.101722    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.101973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.101923    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv78g\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.101973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.101954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102010    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102040    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102106    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102149    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102203    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102270    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102551    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.102710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.102688    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.103362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.103335    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.104435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.104406    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.104752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.104732    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.131145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.131080    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv78g\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.132793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.132755    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.258844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.258814    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:27.504570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.504531    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"]
Oct 30 19:04:27.519421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.516079    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.519421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.519274    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"]
Oct 30 19:04:27.530672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.530329    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:04:27.530672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.530402    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:27.622448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.622409    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"]
Oct 30 19:04:27.630284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.629341    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.634036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.633434    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9wc\" (UniqueName: \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.634036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.633746    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.634036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.633851    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.635425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.635399    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"]
Oct 30 19:04:27.734739 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.734705    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.734912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.734765    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.734912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.734807    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.734912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.734852    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.735335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.735160    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9wc\" (UniqueName: \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.735587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.735557    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc7z\" (UniqueName: \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.735746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.735727    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.737557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.737538    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.762426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.762401    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9wc\" (UniqueName: \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\") pod \"maas-api-revoke-sub-f5c8827c-4sr64\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.768538 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:27.768516    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64930f85_2d31_4da0_8efc_53ab7f4f3721.slice/crio-b513324b85300cfb6e0f0c916982492f46de02532458563ce9f7dd6c016b8a61 WatchSource:0}: Error finding container b513324b85300cfb6e0f0c916982492f46de02532458563ce9f7dd6c016b8a61: Status 404 returned error can't find the container with id b513324b85300cfb6e0f0c916982492f46de02532458563ce9f7dd6c016b8a61
Oct 30 19:04:27.768860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.768840    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:04:27.771165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.771126    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:27.771258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.771201    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:27.771258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.771230    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:27.846911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.844621    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.846911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.844771    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.846911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.844954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc7z\" (UniqueName: \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.846911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.846358    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.849916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.849437    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:27.853732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.853669    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.871395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.870860    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc7z\" (UniqueName: \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\") pod \"maas-api-revoke-sub-d666c3da-4kw8m\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:27.962943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:27.959531    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:28.217552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.217443    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"]
Oct 30 19:04:28.219293 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:28.219255    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c09306_7b14_4e1b_a8a1_9ae9bad9969c.slice/crio-58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f WatchSource:0}: Error finding container 58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f: Status 404 returned error can't find the container with id 58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f
Oct 30 19:04:28.271144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.271078    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64" event={"ID":"51c09306-7b14-4e1b-a8a1-9ae9bad9969c","Type":"ContainerStarted","Data":"58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f"}
Oct 30 19:04:28.273235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.273209    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerStarted","Data":"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"}
Oct 30 19:04:28.273344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.273251    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerStarted","Data":"b513324b85300cfb6e0f0c916982492f46de02532458563ce9f7dd6c016b8a61"}
Oct 30 19:04:28.293590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.293562    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"]
Oct 30 19:04:28.306810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.306722    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podStartSLOduration=2.30670473 podStartE2EDuration="2.30670473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:28.297895204 +0000 UTC m=+1888.561360154" watchObservedRunningTime="2026-10-30 19:04:28.30670473 +0000 UTC m=+1888.570169679"
Oct 30 19:04:28.382913 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:28.382878    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83945599_919f_4a4c_8a7b_d65a09a6da5f.slice/crio-f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2 WatchSource:0}: Error finding container f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2: Status 404 returned error can't find the container with id f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2
Oct 30 19:04:28.879177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.879143    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"]
Oct 30 19:04:28.885210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.885181    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:28.902974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.902946    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"]
Oct 30 19:04:28.981150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.981063    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:28.981589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.981271    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:28.981994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:28.981943    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.083957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.083927    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.084152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.084026    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.084207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.084195    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.084758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.084731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.086683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.086660    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.100473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.100445    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\") pod \"maas-api-revoke-sub-b693a587-kqt2k\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.215881 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.215853    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:29.260449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.260421    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:29.267716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.267682    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:29.292008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.291037    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m" event={"ID":"83945599-919f-4a4c-8a7b-d65a09a6da5f","Type":"ContainerStarted","Data":"8d9ced282c4129a3d1c36f9c69a3ede99463e562ed9a86b6fa239d0c8f4c24eb"}
Oct 30 19:04:29.292008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.291154    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m" event={"ID":"83945599-919f-4a4c-8a7b-d65a09a6da5f","Type":"ContainerStarted","Data":"f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2"}
Oct 30 19:04:29.294775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.294708    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64" event={"ID":"51c09306-7b14-4e1b-a8a1-9ae9bad9969c","Type":"ContainerStarted","Data":"7b8637beb8444e5d5ede0c5b7e21f492dd8818b8b419bb0dfb98e7fd3e9ac288"}
Oct 30 19:04:29.295002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.294988    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:29.298127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.296008    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:04:29.356164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.356078    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64" podStartSLOduration=2.356065193 podStartE2EDuration="2.356065193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:29.351329119 +0000 UTC m=+1889.614794088" watchObservedRunningTime="2026-10-30 19:04:29.356065193 +0000 UTC m=+1889.619530141"
Oct 30 19:04:29.376259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.376137    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m" podStartSLOduration=2.376119797 podStartE2EDuration="2.376119797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:29.372813529 +0000 UTC m=+1889.636278477" watchObservedRunningTime="2026-10-30 19:04:29.376119797 +0000 UTC m=+1889.639584746"
Oct 30 19:04:29.513007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:29.512979    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"]
Oct 30 19:04:30.306076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.306041    5065 generic.go:356] "Generic (PLEG): container finished" podID="51c09306-7b14-4e1b-a8a1-9ae9bad9969c" containerID="7b8637beb8444e5d5ede0c5b7e21f492dd8818b8b419bb0dfb98e7fd3e9ac288" exitCode=0
Oct 30 19:04:30.306541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.306149    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64" event={"ID":"51c09306-7b14-4e1b-a8a1-9ae9bad9969c","Type":"ContainerDied","Data":"7b8637beb8444e5d5ede0c5b7e21f492dd8818b8b419bb0dfb98e7fd3e9ac288"}
Oct 30 19:04:30.308350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.308319    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k" event={"ID":"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a","Type":"ContainerStarted","Data":"816fc2b30a03adeac58b2cbeffa6a7540680c9ada262d2c08d8cc03591e5f769"}
Oct 30 19:04:30.308497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.308374    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k" event={"ID":"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a","Type":"ContainerStarted","Data":"add1968f98a3a0f224473a8b5bd635c8b45f66224c407d91e1a42f6861e044ad"}
Oct 30 19:04:30.310188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.310164    5065 generic.go:356] "Generic (PLEG): container finished" podID="83945599-919f-4a4c-8a7b-d65a09a6da5f" containerID="8d9ced282c4129a3d1c36f9c69a3ede99463e562ed9a86b6fa239d0c8f4c24eb" exitCode=0
Oct 30 19:04:30.310291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.310240    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m" event={"ID":"83945599-919f-4a4c-8a7b-d65a09a6da5f","Type":"ContainerDied","Data":"8d9ced282c4129a3d1c36f9c69a3ede99463e562ed9a86b6fa239d0c8f4c24eb"}
Oct 30 19:04:30.383555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:30.383455    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k" podStartSLOduration=2.3834387 podStartE2EDuration="2.3834387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:30.37930262 +0000 UTC m=+1890.642767570" watchObservedRunningTime="2026-10-30 19:04:30.3834387 +0000 UTC m=+1890.646903649"
Oct 30 19:04:31.322698 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.322664    5065 generic.go:356] "Generic (PLEG): container finished" podID="fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" containerID="816fc2b30a03adeac58b2cbeffa6a7540680c9ada262d2c08d8cc03591e5f769" exitCode=0
Oct 30 19:04:31.323129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.322982    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k" event={"ID":"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a","Type":"ContainerDied","Data":"816fc2b30a03adeac58b2cbeffa6a7540680c9ada262d2c08d8cc03591e5f769"}
Oct 30 19:04:31.531755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.531728    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:31.547922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.547894    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\") pod \"83945599-919f-4a4c-8a7b-d65a09a6da5f\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") "
Oct 30 19:04:31.548112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.547964    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\" (UniqueName: \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\") pod \"83945599-919f-4a4c-8a7b-d65a09a6da5f\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") "
Oct 30 19:04:31.548294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.548272    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\") pod \"83945599-919f-4a4c-8a7b-d65a09a6da5f\" (UID: \"83945599-919f-4a4c-8a7b-d65a09a6da5f\") "
Oct 30 19:04:31.549142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.549115    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca" pod "83945599-919f-4a4c-8a7b-d65a09a6da5f" (UID: "83945599-919f-4a4c-8a7b-d65a09a6da5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:31.551225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.551195    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca" pod "83945599-919f-4a4c-8a7b-d65a09a6da5f" (UID: "83945599-919f-4a4c-8a7b-d65a09a6da5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:31.551990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.551962    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z" pod "83945599-919f-4a4c-8a7b-d65a09a6da5f" (UID: "83945599-919f-4a4c-8a7b-d65a09a6da5f"). InnerVolumeSpecName "kube-api-access-2pc7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:31.632059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.632031    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:31.649533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.649507    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\" (UniqueName: \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\") pod \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") "
Oct 30 19:04:31.649637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.649575    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\") pod \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") "
Oct 30 19:04:31.649719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.649705    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\") pod \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\" (UID: \"51c09306-7b14-4e1b-a8a1-9ae9bad9969c\") "
Oct 30 19:04:31.649934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.649910    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca" pod "51c09306-7b14-4e1b-a8a1-9ae9bad9969c" (UID: "51c09306-7b14-4e1b-a8a1-9ae9bad9969c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:31.650549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.650527    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:31.650649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.650552    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:31.650649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.650562    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pc7z\" (UniqueName: \"kubernetes.io/projected/83945599-919f-4a4c-8a7b-d65a09a6da5f-kube-api-access-2pc7z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:31.650649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.650572    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83945599-919f-4a4c-8a7b-d65a09a6da5f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:31.651770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.651750    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc" pod "51c09306-7b14-4e1b-a8a1-9ae9bad9969c" (UID: "51c09306-7b14-4e1b-a8a1-9ae9bad9969c"). InnerVolumeSpecName "kube-api-access-bx9wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:31.651824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.651785    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca" pod "51c09306-7b14-4e1b-a8a1-9ae9bad9969c" (UID: "51c09306-7b14-4e1b-a8a1-9ae9bad9969c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:31.751791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.751760    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9wc\" (UniqueName: \"kubernetes.io/projected/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-kube-api-access-bx9wc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:31.751791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:31.751786    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c09306-7b14-4e1b-a8a1-9ae9bad9969c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:32.482979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.482943    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m" event={"ID":"83945599-919f-4a4c-8a7b-d65a09a6da5f","Type":"ContainerDied","Data":"f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2"}
Oct 30 19:04:32.483573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.483553    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ce7f38ced7f590d6cde8b390f808bba98e36e9bf914ded1d54cd70cdff4fe2"
Oct 30 19:04:32.483734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.483722    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"
Oct 30 19:04:32.494799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.494773    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"
Oct 30 19:04:32.495739 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.495711    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64" event={"ID":"51c09306-7b14-4e1b-a8a1-9ae9bad9969c","Type":"ContainerDied","Data":"58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f"}
Oct 30 19:04:32.495926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.495912    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58981c478851f1d2fe4d712363ad9436ed1cec3b4fa0283bf04c63392cb2032f"
Oct 30 19:04:32.765307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.765279    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:32.865661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.865633    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\") pod \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") "
Oct 30 19:04:32.865829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.865698    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\") pod \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") "
Oct 30 19:04:32.865927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.865915    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\") pod \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\" (UID: \"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a\") "
Oct 30 19:04:32.866540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.866505    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca" pod "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" (UID: "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:32.868032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.867963    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca" pod "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" (UID: "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:32.868383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.868362    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm" pod "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" (UID: "fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a"). InnerVolumeSpecName "kube-api-access-xdknm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:32.967663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.967635    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:32.967663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.967664    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:32.967663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:32.967674    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdknm\" (UniqueName: \"kubernetes.io/projected/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a-kube-api-access-xdknm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:33.514828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:33.514782    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k" event={"ID":"fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a","Type":"ContainerDied","Data":"add1968f98a3a0f224473a8b5bd635c8b45f66224c407d91e1a42f6861e044ad"}
Oct 30 19:04:33.515342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:33.514898    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add1968f98a3a0f224473a8b5bd635c8b45f66224c407d91e1a42f6861e044ad"
Oct 30 19:04:33.515342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:33.514994    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"
Oct 30 19:04:39.213530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:39.213490    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 19:04:39.214077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:39.213733    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy" containerID="cri-o://668b07d0b94c45d04c0cbc3de938bf30b87c850a9427c8b355ee6a6b88ec598f" gracePeriod=30
Oct 30 19:04:39.785892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:39.778853    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:04:42.371824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:42.370166    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 19:04:42.371824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:42.370439    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy" containerID="cri-o://5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e" gracePeriod=30
Oct 30 19:04:42.377505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:42.374763    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 19:04:42.377505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:42.374950    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy" containerID="cri-o://6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a" gracePeriod=30
Oct 30 19:04:42.477069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:42.476493    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:04:44.583119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:44.581790    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 19:04:44.583119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:44.582021    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy" containerID="cri-o://6ac24efb998539c7e8310ac51e916b26a42b2dc28840ca1fe2cd67553058fb13" gracePeriod=30
Oct 30 19:04:45.047752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.046048    5065 generic.go:356] "Generic (PLEG): container finished" podID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerID="668b07d0b94c45d04c0cbc3de938bf30b87c850a9427c8b355ee6a6b88ec598f" exitCode=0
Oct 30 19:04:45.047752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.046195    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" event={"ID":"487b6a41-850f-4f31-8e00-aedc152e04ee","Type":"ContainerDied","Data":"668b07d0b94c45d04c0cbc3de938bf30b87c850a9427c8b355ee6a6b88ec598f"}
Oct 30 19:04:45.053185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.053039    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 19:04:45.130184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.129760    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.129888    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130027    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130149    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130232    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130276    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130333    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130364    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.130748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.130416    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\") pod \"487b6a41-850f-4f31-8e00-aedc152e04ee\" (UID: \"487b6a41-850f-4f31-8e00-aedc152e04ee\") "
Oct 30 19:04:45.131371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.131255    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:45.131466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.131430    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:45.132189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.132048    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:45.132189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.132110    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:45.133696 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.133600    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:45.134877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.134839    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:45.135195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.135168    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "kube-api-access-k5hbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:45.136631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.136603    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:45.137769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.137736    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy" pod "487b6a41-850f-4f31-8e00-aedc152e04ee" (UID: "487b6a41-850f-4f31-8e00-aedc152e04ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232285    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487b6a41-850f-4f31-8e00-aedc152e04ee-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232313    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232323    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5hbr\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-kube-api-access-k5hbr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232333    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232341    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232350    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232358    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487b6a41-850f-4f31-8e00-aedc152e04ee-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232367    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:45.232688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:45.232375    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487b6a41-850f-4f31-8e00-aedc152e04ee-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:46.057565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.057536    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"
Oct 30 19:04:46.057982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.057566    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r" event={"ID":"487b6a41-850f-4f31-8e00-aedc152e04ee","Type":"ContainerDied","Data":"f30209aa590d0fd30297153e4e7bf54e3761979b355c072ade6f89e6c09838f8"}
Oct 30 19:04:46.057982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.057627    5065 scope.go:122] "RemoveContainer" containerID="668b07d0b94c45d04c0cbc3de938bf30b87c850a9427c8b355ee6a6b88ec598f"
Oct 30 19:04:46.103862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.103831    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 19:04:46.135340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.135311    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6ms9r"]
Oct 30 19:04:46.930415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.930378    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 19:04:46.931516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.931235    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy" containerID="cri-o://0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c" gracePeriod=30
Oct 30 19:04:46.937575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.937397    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 19:04:46.937731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:46.937655    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy" containerID="cri-o://6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c" gracePeriod=30
Oct 30 19:04:47.867170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:47.866779    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 19:04:48.004637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.004604    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.004935 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.004917    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.012229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.012201    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.012453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.012436    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.012604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.012590    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.012780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.012764    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.013003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.012970    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.013340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.013322    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.018298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.018276    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\") pod \"c79b9022-78a9-434b-bc78-726c3ba4ce61\" (UID: \"c79b9022-78a9-434b-bc78-726c3ba4ce61\") "
Oct 30 19:04:48.022502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.013913    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 19:04:48.023323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.013610    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.023434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.017158    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:48.023505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.018622    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.027213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.018875    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:48.027213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.020486    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.027213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.021777    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "kube-api-access-k4fnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:48.027856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.027770    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.029410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.029391    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" path="/var/lib/kubelet/pods/487b6a41-850f-4f31-8e00-aedc152e04ee/volumes"
Oct 30 19:04:48.040536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.038527    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.040536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.040164    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token" pod "c79b9022-78a9-434b-bc78-726c3ba4ce61" (UID: "c79b9022-78a9-434b-bc78-726c3ba4ce61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:48.108814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.108786    5065 generic.go:356] "Generic (PLEG): container finished" podID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerID="6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a" exitCode=0
Oct 30 19:04:48.109177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.108916    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" event={"ID":"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82","Type":"ContainerDied","Data":"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"}
Oct 30 19:04:48.109378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.109361    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5" event={"ID":"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82","Type":"ContainerDied","Data":"d996d45c15cd6073d981fd8acadcbf888776185460ef6605e8cdc88b539b97b5"}
Oct 30 19:04:48.109514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.109497    5065 scope.go:122] "RemoveContainer" containerID="6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"
Oct 30 19:04:48.109701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.109012    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"
Oct 30 19:04:48.111642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.111625    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"
Oct 30 19:04:48.112284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.111579    5065 generic.go:356] "Generic (PLEG): container finished" podID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerID="5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e" exitCode=0
Oct 30 19:04:48.112387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.111644    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" event={"ID":"c79b9022-78a9-434b-bc78-726c3ba4ce61","Type":"ContainerDied","Data":"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"}
Oct 30 19:04:48.112458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.112386    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm" event={"ID":"c79b9022-78a9-434b-bc78-726c3ba4ce61","Type":"ContainerDied","Data":"626607ede1ad4f3471a434428df79244dbcb9f862d4e77167bb66baef922abe0"}
Oct 30 19:04:48.123132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123066    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123137    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123189    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123272    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123392    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123467    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123526    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123556    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.123718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.123702    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\") pod \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\" (UID: \"ec5bfa6d-07d8-4c0b-a558-b6322de9eb82\") "
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124266    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4fnp\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-kube-api-access-k4fnp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124290    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124307    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124323    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124338    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124353    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79b9022-78a9-434b-bc78-726c3ba4ce61-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124369    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124384    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79b9022-78a9-434b-bc78-726c3ba4ce61-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.127607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.124400    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79b9022-78a9-434b-bc78-726c3ba4ce61-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.130678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.129939    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.130678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.130157    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.131144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.130894    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.131920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.131731    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:48.135582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.133157    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.136311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.136274    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:48.136560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.136527    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:48.136812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.136775    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "kube-api-access-w7trb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:48.140603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.140581    5065 scope.go:122] "RemoveContainer" containerID="6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"
Oct 30 19:04:48.142159 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:48.141364    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a\": container with ID starting with 6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a not found: ID does not exist" containerID="6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"
Oct 30 19:04:48.142159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.141407    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a"} err="failed to get container status \"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a\": rpc error: code = NotFound desc = could not find container \"6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a\": container with ID starting with 6ea876d690ad867e69b6888031e01a4621a5d0d3a1b2b4dcab1903bb1d6f5d5a not found: ID does not exist"
Oct 30 19:04:48.142159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.141430    5065 scope.go:122] "RemoveContainer" containerID="5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"
Oct 30 19:04:48.146031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.145993    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo" pod "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" (UID: "ec5bfa6d-07d8-4c0b-a558-b6322de9eb82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:48.161138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.160224    5065 scope.go:122] "RemoveContainer" containerID="5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"
Oct 30 19:04:48.161138 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:48.160631    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e\": container with ID starting with 5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e not found: ID does not exist" containerID="5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"
Oct 30 19:04:48.161138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.160665    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e"} err="failed to get container status \"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e\": rpc error: code = NotFound desc = could not find container \"5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e\": container with ID starting with 5a9f56b756a044dc03fc0912aea83e4aae7ba055950a9595c55a59a3eae7466e not found: ID does not exist"
Oct 30 19:04:48.164444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.164422    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 19:04:48.180381 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.180353    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-blcfm"]
Oct 30 19:04:48.226167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226136    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226616    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226647    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226661    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226676    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226689    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7trb\" (UniqueName: \"kubernetes.io/projected/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-kube-api-access-w7trb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226716    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226729    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.226866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.226743    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:48.623111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.619141    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 19:04:48.655231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:48.643980    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-dwvm5"]
Oct 30 19:04:49.965713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:49.965675    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" path="/var/lib/kubelet/pods/c79b9022-78a9-434b-bc78-726c3ba4ce61/volumes"
Oct 30 19:04:49.966427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:49.966407    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" path="/var/lib/kubelet/pods/ec5bfa6d-07d8-4c0b-a558-b6322de9eb82/volumes"
Oct 30 19:04:50.040181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.039266    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:50.040181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.039334    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:50.049706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.049668    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:04:50.050702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050677    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy"
Oct 30 19:04:50.050702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050702    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050725    5065 cpu_manager.go:414] "containerMap: removing container" podUID="51c09306-7b14-4e1b-a8a1-9ae9bad9969c" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050732    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c09306-7b14-4e1b-a8a1-9ae9bad9969c" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050755    5065 cpu_manager.go:414] "containerMap: removing container" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050761    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050808    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050815    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050836    5065 cpu_manager.go:414] "containerMap: removing container" podUID="83945599-919f-4a4c-8a7b-d65a09a6da5f" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050843    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83945599-919f-4a4c-8a7b-d65a09a6da5f" containerName="revoke-keys"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050860    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy"
Oct 30 19:04:50.050895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.050867    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051069    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" containerName="revoke-keys"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051102    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="83945599-919f-4a4c-8a7b-d65a09a6da5f" containerName="revoke-keys"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051114    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79b9022-78a9-434b-bc78-726c3ba4ce61" containerName="istio-proxy"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051129    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5bfa6d-07d8-4c0b-a558-b6322de9eb82" containerName="istio-proxy"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051143    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c09306-7b14-4e1b-a8a1-9ae9bad9969c" containerName="revoke-keys"
Oct 30 19:04:50.051491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.051153    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="487b6a41-850f-4f31-8e00-aedc152e04ee" containerName="istio-proxy"
Oct 30 19:04:50.073700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.073650    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.111053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.110962    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c9e9de-openshift-default-dockercfg-ct2zv\""
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.138867    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.138922    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.138986    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139013    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139255    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139283    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139335    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2hp\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139392    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.139946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.139591    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2hp\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242291    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242357    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242399    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242469    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242495    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242536    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242780    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.242812    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.243392    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.263111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.244778    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.433996    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.568413    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.584201    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2hp\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.603227    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.603454    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.622847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.604527    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.681558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.680886    5065 generic.go:356] "Generic (PLEG): container finished" podID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerID="6ac24efb998539c7e8310ac51e916b26a42b2dc28840ca1fe2cd67553058fb13" exitCode=0
Oct 30 19:04:50.681558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.680937    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" event={"ID":"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0","Type":"ContainerDied","Data":"6ac24efb998539c7e8310ac51e916b26a42b2dc28840ca1fe2cd67553058fb13"}
Oct 30 19:04:50.718475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.718146    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:04:50.725319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.725293    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:50.730563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.730477    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:04:50.756015 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.755966    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e8c554-dockercfg-fcg4f\""
Oct 30 19:04:50.756271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.756123    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e8c554\""
Oct 30 19:04:50.820110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.819376    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:50.898846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.869202    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:04:50.944169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.942637    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:50.972444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.971957    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfpp\" (UniqueName: \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:50.972444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:50.972260    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.061746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.052412    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:51.078787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.073645    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.078787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.073689    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.078787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.073737    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgmb\" (UniqueName: \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.078787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.073890    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfpp\" (UniqueName: \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.107029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.085725    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.189837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.189741    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.190118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.190071    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgmb\" (UniqueName: \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.213118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.191022    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.213118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.193781    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:04:51.213118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.197479    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfpp\" (UniqueName: \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\") pod \"payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.232869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.232838    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgmb\" (UniqueName: \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\") pod \"payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.250513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.250488    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:51.259834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.259813    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 19:04:51.334801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.333383    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393499    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393579    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393626    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393700    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393746    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393829    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393868    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.393898    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.394522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.394038    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\") pod \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\" (UID: \"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0\") "
Oct 30 19:04:51.398151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.397205    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:51.398151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.397510    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:51.398151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.397574    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:51.398151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.398112    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:51.398963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.398941    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:51.408810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.408769    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "kube-api-access-4b4hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:51.464211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.463941    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:51.465871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.465547    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:51.466929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.466734    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo" pod "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" (UID: "153c22a8-3d5a-4152-8e8a-1ab9e05bdff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497565    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497598    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497613    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497627    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497641    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497655    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497668    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497681    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.497721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.497695    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4hh\" (UniqueName: \"kubernetes.io/projected/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0-kube-api-access-4b4hh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:51.639015 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.638657    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:04:51.643792 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:51.643668    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e83d2a2_f6fa_4a6b_bd78_1c947eaf1a5f.slice/crio-ff5bd4a5456fed889677290fe0938f7e05a877773fd6bfc48cc9e005dc948eca WatchSource:0}: Error finding container ff5bd4a5456fed889677290fe0938f7e05a877773fd6bfc48cc9e005dc948eca: Status 404 returned error can't find the container with id ff5bd4a5456fed889677290fe0938f7e05a877773fd6bfc48cc9e005dc948eca
Oct 30 19:04:51.646596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.646563    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:51.646715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.646635    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:51.646715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.646679    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:51.707659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.705464    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerStarted","Data":"ff5bd4a5456fed889677290fe0938f7e05a877773fd6bfc48cc9e005dc948eca"}
Oct 30 19:04:51.711788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.709963    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs" event={"ID":"153c22a8-3d5a-4152-8e8a-1ab9e05bdff0","Type":"ContainerDied","Data":"aaefb2703f9ccaec419c900e3b06d595a87bc0fa689201dc819e62abba65820a"}
Oct 30 19:04:51.711788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.710026    5065 scope.go:122] "RemoveContainer" containerID="6ac24efb998539c7e8310ac51e916b26a42b2dc28840ca1fe2cd67553058fb13"
Oct 30 19:04:51.711788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.710190    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"
Oct 30 19:04:51.754307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.754277    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 19:04:51.759586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.759560    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-582xs"]
Oct 30 19:04:51.774832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.774801    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:04:51.814244 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:51.814205    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0710a88e_b40d_4031_99ee_9ab808e382f5.slice/crio-ac4a77c958de5634189f09a8ab59f495bab8d3f538a04112a8a80d19ba96b1a1 WatchSource:0}: Error finding container ac4a77c958de5634189f09a8ab59f495bab8d3f538a04112a8a80d19ba96b1a1: Status 404 returned error can't find the container with id ac4a77c958de5634189f09a8ab59f495bab8d3f538a04112a8a80d19ba96b1a1
Oct 30 19:04:51.835278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:51.835256    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:04:51.844193 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:51.844165    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7d5e5d_dd35_422e_83d8_f97b120a101a.slice/crio-6f8cc82455f2dc2f244836616f5c632f0b7fbdb90c5f03887f0c2ba020fc6c42 WatchSource:0}: Error finding container 6f8cc82455f2dc2f244836616f5c632f0b7fbdb90c5f03887f0c2ba020fc6c42: Status 404 returned error can't find the container with id 6f8cc82455f2dc2f244836616f5c632f0b7fbdb90c5f03887f0c2ba020fc6c42
Oct 30 19:04:52.050217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.034260    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" path="/var/lib/kubelet/pods/153c22a8-3d5a-4152-8e8a-1ab9e05bdff0/volumes"
Oct 30 19:04:52.363895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.363867    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 19:04:52.373455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.373431    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 19:04:52.444911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.444876    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.444927    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.444962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445063    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445149    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445207    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445261    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445308    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445362    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445440    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445485    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445523    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445561    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445595    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445653    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445703    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445765    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\") pod \"19a6e8be-e060-4dab-9347-466160f4c3b8\" (UID: \"19a6e8be-e060-4dab-9347-466160f4c3b8\") "
Oct 30 19:04:52.447175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.445911    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\") pod \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\" (UID: \"68a6722d-13ca-40ef-91a4-5730fa7f88ab\") "
Oct 30 19:04:52.448230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.447517    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.448325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.448301    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.449367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.449164    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.449683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.449533    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.449930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.449791    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.450217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.450029    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.450653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.450399    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.451512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.450898    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:52.451652 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.451474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.451791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.451770    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:52.452284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.452265    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:52.454604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.453700    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "kube-api-access-r4pd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:52.454604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.454438    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.454604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.454578    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:52.454883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.454859    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo" pod "68a6722d-13ca-40ef-91a4-5730fa7f88ab" (UID: "68a6722d-13ca-40ef-91a4-5730fa7f88ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:52.455157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.455063    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:52.455389 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.455367    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:52.458842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.458780    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt" pod "19a6e8be-e060-4dab-9347-466160f4c3b8" (UID: "19a6e8be-e060-4dab-9347-466160f4c3b8"). InnerVolumeSpecName "kube-api-access-km5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:52.551820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.551782    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552078 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552061    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552202    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552305    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552327    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552342    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5wt\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-kube-api-access-km5wt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552357    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552371    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552384    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552399    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552412    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pd9\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-kube-api-access-r4pd9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552425    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552438    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552496    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.552541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552511    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a6e8be-e060-4dab-9347-466160f4c3b8-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.553269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552549    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68a6722d-13ca-40ef-91a4-5730fa7f88ab-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.553269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552563    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68a6722d-13ca-40ef-91a4-5730fa7f88ab-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.553269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.552577    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a6e8be-e060-4dab-9347-466160f4c3b8-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:52.721329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.721231    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" event={"ID":"0710a88e-b40d-4031-99ee-9ab808e382f5","Type":"ContainerStarted","Data":"f41631affb75a2d39a1ec4db2675ed263d19127db9fb20e8fbffd3c688ce523c"}
Oct 30 19:04:52.721329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.721288    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" event={"ID":"0710a88e-b40d-4031-99ee-9ab808e382f5","Type":"ContainerStarted","Data":"ac4a77c958de5634189f09a8ab59f495bab8d3f538a04112a8a80d19ba96b1a1"}
Oct 30 19:04:52.721329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.721315    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:04:52.723318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.723291    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" event={"ID":"da7d5e5d-dd35-422e-83d8-f97b120a101a","Type":"ContainerStarted","Data":"348a901b7d0d3b50b1d5fd067576bb2783466006c3942d6dd26625816295e3a2"}
Oct 30 19:04:52.723465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.723339    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" event={"ID":"da7d5e5d-dd35-422e-83d8-f97b120a101a","Type":"ContainerStarted","Data":"6f8cc82455f2dc2f244836616f5c632f0b7fbdb90c5f03887f0c2ba020fc6c42"}
Oct 30 19:04:52.723465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.723367    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:04:52.725055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.725029    5065 generic.go:356] "Generic (PLEG): container finished" podID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerID="0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c" exitCode=0
Oct 30 19:04:52.725204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.725074    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"
Oct 30 19:04:52.725204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.725127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" event={"ID":"19a6e8be-e060-4dab-9347-466160f4c3b8","Type":"ContainerDied","Data":"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"}
Oct 30 19:04:52.725204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.725180    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr" event={"ID":"19a6e8be-e060-4dab-9347-466160f4c3b8","Type":"ContainerDied","Data":"6e3edb995798f5c94dd1ded8b84f4887b30430c0d3983fd2d6586a7f97d90967"}
Oct 30 19:04:52.725384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.725212    5065 scope.go:122] "RemoveContainer" containerID="0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"
Oct 30 19:04:52.729975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.729952    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerStarted","Data":"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"}
Oct 30 19:04:52.731818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.731788    5065 generic.go:356] "Generic (PLEG): container finished" podID="a047c669-648f-4d9b-86ef-866441a1330b" containerID="52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac" exitCode=0
Oct 30 19:04:52.731942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.731862    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerDied","Data":"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"}
Oct 30 19:04:52.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.732293    5065 scope.go:122] "RemoveContainer" containerID="52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"
Oct 30 19:04:52.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.734602    5065 generic.go:356] "Generic (PLEG): container finished" podID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerID="6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c" exitCode=0
Oct 30 19:04:52.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.734679    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" event={"ID":"68a6722d-13ca-40ef-91a4-5730fa7f88ab","Type":"ContainerDied","Data":"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"}
Oct 30 19:04:52.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.734713    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz" event={"ID":"68a6722d-13ca-40ef-91a4-5730fa7f88ab","Type":"ContainerDied","Data":"18faff16741de63a1d27802b32e8565935d2cae653c44a5dddcecaf63b7eee44"}
Oct 30 19:04:52.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.734772    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"
Oct 30 19:04:52.744537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.744492    5065 scope.go:122] "RemoveContainer" containerID="0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"
Oct 30 19:04:52.745399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745355    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" podStartSLOduration=2.745342147 podStartE2EDuration="2.745342147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:52.745103032 +0000 UTC m=+1913.008567990" watchObservedRunningTime="2026-10-30 19:04:52.745342147 +0000 UTC m=+1913.008807096"
Oct 30 19:04:52.745639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745604    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:52.745714 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:52.745466    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c\": container with ID starting with 0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c not found: ID does not exist" containerID="0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"
Oct 30 19:04:52.745714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745703    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:52.745830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745758    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:04:52.745830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745768    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c"} err="failed to get container status \"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c\": rpc error: code = NotFound desc = could not find container \"0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c\": container with ID starting with 0c5aeb8b5e9aa98d35a38d72012650ea4d4743b13dbfde18cd2c7463b96ac32c not found: ID does not exist"
Oct 30 19:04:52.745830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.745789    5065 scope.go:122] "RemoveContainer" containerID="6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"
Oct 30 19:04:52.766232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.766208    5065 scope.go:122] "RemoveContainer" containerID="6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"
Oct 30 19:04:52.766531 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:52.766511    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c\": container with ID starting with 6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c not found: ID does not exist" containerID="6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"
Oct 30 19:04:52.766717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.766541    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c"} err="failed to get container status \"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c\": rpc error: code = NotFound desc = could not find container \"6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c\": container with ID starting with 6c763902df9432fd35e5fc9087b9e46e6efe7d644d64fcbc31214a8a7c37194c not found: ID does not exist"
Oct 30 19:04:52.801903 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.801843    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podStartSLOduration=2.801826016 podStartE2EDuration="2.801826016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:52.790770308 +0000 UTC m=+1913.054235257" watchObservedRunningTime="2026-10-30 19:04:52.801826016 +0000 UTC m=+1913.065290964"
Oct 30 19:04:52.822667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.822617    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 19:04:52.828965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.828935    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-vn9dr"]
Oct 30 19:04:52.915375 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.915322    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" podStartSLOduration=2.915304643 podStartE2EDuration="2.915304643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:52.886938614 +0000 UTC m=+1913.150403592" watchObservedRunningTime="2026-10-30 19:04:52.915304643 +0000 UTC m=+1913.178769593"
Oct 30 19:04:52.940743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.940708    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 19:04:52.945995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:52.945964    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-wcqrz"]
Oct 30 19:04:53.068120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.067538    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:53.084629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.084363    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:53.135820 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:04:53.135780    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:04:53.747104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.747060    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerStarted","Data":"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"}
Oct 30 19:04:53.749503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.749478    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:53.750277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.750257    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:04:53.963305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.963270    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" path="/var/lib/kubelet/pods/19a6e8be-e060-4dab-9347-466160f4c3b8/volumes"
Oct 30 19:04:53.964105 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:53.964066    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" path="/var/lib/kubelet/pods/68a6722d-13ca-40ef-91a4-5730fa7f88ab/volumes"
Oct 30 19:04:54.199110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:54.199060    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:04:54.199790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:54.199767    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:54.199964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:54.199935    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:55.196468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.196432    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:55.196841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.196811    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:55.228934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.228900    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" podUID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" containerName="main" containerID="cri-o://2f6b1a8ea8651a0e1c2ee1c041d8ac7619d27d64d7d7b741e1dd0dbaf27d82d2" gracePeriod=45
Oct 30 19:04:55.834174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.834114    5065 generic.go:356] "Generic (PLEG): container finished" podID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" containerID="2f6b1a8ea8651a0e1c2ee1c041d8ac7619d27d64d7d7b741e1dd0dbaf27d82d2" exitCode=0
Oct 30 19:04:55.847260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.835341    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" event={"ID":"9b950233-52f8-4bb4-99dd-6760d95bc6a6","Type":"ContainerDied","Data":"2f6b1a8ea8651a0e1c2ee1c041d8ac7619d27d64d7d7b741e1dd0dbaf27d82d2"}
Oct 30 19:04:55.938011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:55.937984    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.025716    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\" (UniqueName: \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.025797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.025846    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.025879    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.025940    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.026061    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\") pod \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\" (UID: \"9b950233-52f8-4bb4-99dd-6760d95bc6a6\") "
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.026665    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.026858    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.027283    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.027293    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:56.029013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.027303    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.031143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.029947    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:56.031143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.030065    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:56.031787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.031761    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l" pod "9b950233-52f8-4bb4-99dd-6760d95bc6a6" (UID: "9b950233-52f8-4bb4-99dd-6760d95bc6a6"). InnerVolumeSpecName "kube-api-access-9gc2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:56.128746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.128661    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gc2l\" (UniqueName: \"kubernetes.io/projected/9b950233-52f8-4bb4-99dd-6760d95bc6a6-kube-api-access-9gc2l\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.128746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.128690    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b950233-52f8-4bb4-99dd-6760d95bc6a6-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.128746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.128701    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.128746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.128710    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b950233-52f8-4bb4-99dd-6760d95bc6a6-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:56.198475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.198439    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:56.198744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.198716    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:56.711760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.711724    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:04:56.712825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712800    5065 cpu_manager.go:414] "containerMap: removing container" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy"
Oct 30 19:04:56.712825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712822    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712849    5065 cpu_manager.go:414] "containerMap: removing container" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712856    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712881    5065 cpu_manager.go:414] "containerMap: removing container" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712888    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712910    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" containerName="main"
Oct 30 19:04:56.713018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.712916    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" containerName="main"
Oct 30 19:04:56.713390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.713120    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a6e8be-e060-4dab-9347-466160f4c3b8" containerName="istio-proxy"
Oct 30 19:04:56.713390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.713139    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c22a8-3d5a-4152-8e8a-1ab9e05bdff0" containerName="istio-proxy"
Oct 30 19:04:56.713390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.713153    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a6722d-13ca-40ef-91a4-5730fa7f88ab" containerName="istio-proxy"
Oct 30 19:04:56.713390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.713167    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" containerName="main"
Oct 30 19:04:56.718299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.718271    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:04:56.718485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.718453    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:56.724784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.724759    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:04:56.724915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.724860    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:56.738526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.738500    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:04:56.740436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.738861    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c9e9de\""
Oct 30 19:04:56.805437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.805403    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c9e9de-dockercfg-mgs6m\""
Oct 30 19:04:56.916114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.915518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8s22\" (UniqueName: \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:56.916114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.915643    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:56.916114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.915718    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:56.916114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.915931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57xj\" (UniqueName: \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:56.996325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.996234    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775" event={"ID":"9b950233-52f8-4bb4-99dd-6760d95bc6a6","Type":"ContainerDied","Data":"205af4728dd36829323b1b24f01365e59cdf6c6ac2e82e15b8f403074e5e65e0"}
Oct 30 19:04:56.996577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.996556    5065 scope.go:122] "RemoveContainer" containerID="2f6b1a8ea8651a0e1c2ee1c041d8ac7619d27d64d7d7b741e1dd0dbaf27d82d2"
Oct 30 19:04:56.996794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:56.996592    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"
Oct 30 19:04:57.027496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.027440    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8s22\" (UniqueName: \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:57.029290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.029248    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:57.029499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.028053    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:57.029715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.029697    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:57.049768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.049733    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57xj\" (UniqueName: \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:57.050442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.050418    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:57.171120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.166332    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:04:57.171329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.171243    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c3a5b9/auth-test-model-c3a5b9-kserve-5587bf88d5-g8775"]
Oct 30 19:04:57.173758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.173668    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8s22\" (UniqueName: \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\") pod \"payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:57.173758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.173667    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57xj\" (UniqueName: \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\") pod \"payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") " pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:57.212823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.212351    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:57.212823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.212411    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:57.350341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.350296    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:57.364184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.364156    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:57.562729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.562694    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main" containerID="cri-o://e5b5788c670778a7726b4a53e73516674ecaed57a7be1112d45c56a20bd9e8f5" gracePeriod=45
Oct 30 19:04:57.836473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.836352    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:04:57.847004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.846296    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:04:57.852356 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:04:57.852332    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294bf429_04b4_414b_9c6f_9dd466ea723b.slice/crio-9b57189cf089263f322f1402c1d6a053d963c36ff332f9d07e14f76e3441f3c9 WatchSource:0}: Error finding container 9b57189cf089263f322f1402c1d6a053d963c36ff332f9d07e14f76e3441f3c9: Status 404 returned error can't find the container with id 9b57189cf089263f322f1402c1d6a053d963c36ff332f9d07e14f76e3441f3c9
Oct 30 19:04:57.907212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.871189    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.255:8000/ready\": dial tcp 10.132.0.255:8000: connect: connection refused"
Oct 30 19:04:57.976766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:57.976739    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b950233-52f8-4bb4-99dd-6760d95bc6a6" path="/var/lib/kubelet/pods/9b950233-52f8-4bb4-99dd-6760d95bc6a6/volumes"
Oct 30 19:04:58.039757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.037284    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" event={"ID":"c76d05b0-f43a-480b-8d7c-627f6d37d8ad","Type":"ContainerStarted","Data":"86c82fd2266f442802707015fde58eae493ce19b2abb097670c90ea3a01a7698"}
Oct 30 19:04:58.041074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.041044    5065 generic.go:356] "Generic (PLEG): container finished" podID="e10a5317-02a3-4949-a47b-f2532542c40d" containerID="e5b5788c670778a7726b4a53e73516674ecaed57a7be1112d45c56a20bd9e8f5" exitCode=0
Oct 30 19:04:58.041585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.041557    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" event={"ID":"e10a5317-02a3-4949-a47b-f2532542c40d","Type":"ContainerDied","Data":"e5b5788c670778a7726b4a53e73516674ecaed57a7be1112d45c56a20bd9e8f5"}
Oct 30 19:04:58.045626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.045602    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" event={"ID":"294bf429-04b4-414b-9c6f-9dd466ea723b","Type":"ContainerStarted","Data":"9b57189cf089263f322f1402c1d6a053d963c36ff332f9d07e14f76e3441f3c9"}
Oct 30 19:04:58.213254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.211632    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:58.213254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.211711    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:58.418492 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.418026    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.452802    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.452962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453014    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453045    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453071    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.453154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453125    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\" (UniqueName: \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\") pod \"e10a5317-02a3-4949-a47b-f2532542c40d\" (UID: \"e10a5317-02a3-4949-a47b-f2532542c40d\") "
Oct 30 19:04:58.456578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453453    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:58.456578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.453972    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:58.456578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.454242    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:58.463647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.463017    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:58.463647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.463489    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:58.474766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.474698    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "kube-api-access-45glc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:58.475517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.475464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm" pod "e10a5317-02a3-4949-a47b-f2532542c40d" (UID: "e10a5317-02a3-4949-a47b-f2532542c40d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:58.555376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.555331    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:58.555376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.555371    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10a5317-02a3-4949-a47b-f2532542c40d-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:58.555376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.555383    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:58.555720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.555398    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10a5317-02a3-4949-a47b-f2532542c40d-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:58.555720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:58.555410    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45glc\" (UniqueName: \"kubernetes.io/projected/e10a5317-02a3-4949-a47b-f2532542c40d-kube-api-access-45glc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:59.058318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.058273    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" event={"ID":"294bf429-04b4-414b-9c6f-9dd466ea723b","Type":"ContainerStarted","Data":"ea7373657459d5482d6216f9b97434138a1ad0a204f332d016a4b013ee6e1f35"}
Oct 30 19:04:59.058805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.058361    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:04:59.060512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.060472    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" event={"ID":"c76d05b0-f43a-480b-8d7c-627f6d37d8ad","Type":"ContainerStarted","Data":"319aca329c3490400954bd18402ee048972ddb9acb189949da6ff4de74f28cf4"}
Oct 30 19:04:59.060662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.060557    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:04:59.062382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.062355    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr" event={"ID":"e10a5317-02a3-4949-a47b-f2532542c40d","Type":"ContainerDied","Data":"430ffeaac5290b789e59ca00c0fa3bb97e4d655750227b046a56307f3ccb0ac2"}
Oct 30 19:04:59.062687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.062665    5065 scope.go:122] "RemoveContainer" containerID="e5b5788c670778a7726b4a53e73516674ecaed57a7be1112d45c56a20bd9e8f5"
Oct 30 19:04:59.062796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.062377    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"
Oct 30 19:04:59.090186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.089412    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" podStartSLOduration=3.089377602 podStartE2EDuration="3.089377602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:59.078983792 +0000 UTC m=+1919.342448751" watchObservedRunningTime="2026-10-30 19:04:59.089377602 +0000 UTC m=+1919.352842551"
Oct 30 19:04:59.121944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.121914    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:04:59.126820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.126788    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03362d/auth-test-model-03362d-kserve-6c4df564fc-qt5kr"]
Oct 30 19:04:59.143468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.143402    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" podStartSLOduration=3.143385202 podStartE2EDuration="3.143385202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:59.14099658 +0000 UTC m=+1919.404461544" watchObservedRunningTime="2026-10-30 19:04:59.143385202 +0000 UTC m=+1919.406850149"
Oct 30 19:04:59.195945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.195904    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 19:04:59.196224 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.195969    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 19:04:59.894806 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:04:59.894763    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" path="/var/lib/kubelet/pods/e10a5317-02a3-4949-a47b-f2532542c40d/volumes"
Oct 30 19:05:00.038109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.038059    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:05:00.039483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.039451    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main"
Oct 30 19:05:00.039483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.039477    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main"
Oct 30 19:05:00.039682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.039646    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10a5317-02a3-4949-a47b-f2532542c40d" containerName="main"
Oct 30 19:05:00.045055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.045024    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.061244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.059916    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-de4bda-openshift-default-dockercfg-jchfn\""
Oct 30 19:05:00.066556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.066514    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:05:00.081913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.081877    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.081953    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082022    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082146    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082216    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082251    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082417    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkqb\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082503    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.082602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.082561    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.200630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.200525    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkqb\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.246008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.245974    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.246337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.246310    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.246526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.246508    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.246713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.246696    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.247187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.246809    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.247397    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.247487    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.247597    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.247715    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.247756    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.248273    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.248742    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.259113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.248977    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.294111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.285317    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkqb\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.294562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.288603    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.294709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.294675    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.311785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.311708    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.401112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.396289    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-de4bda-openshift-default-dockercfg-jchfn\""
Oct 30 19:05:00.401686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.401409    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:00.416113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.409975    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:05:00.416368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.416337    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:05:00.421140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.420929    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:05:00.882530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.882488    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:05:00.887371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.887322    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:00.887525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.887406    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:00.887525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:00.887438    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:01.119109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.118613    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerStarted","Data":"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"}
Oct 30 19:05:01.119109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.118870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerStarted","Data":"86b222059b26192daef3181571b816347da745f3dbbc64aa042366a9ee5c36f6"}
Oct 30 19:05:01.157967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.157861    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podStartSLOduration=1.157833016 podStartE2EDuration="1.157833016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:01.156116485 +0000 UTC m=+1921.419581434" watchObservedRunningTime="2026-10-30 19:05:01.157833016 +0000 UTC m=+1921.421297964"
Oct 30 19:05:01.405567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.405527    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:01.407318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.407284    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:05:01.407494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:01.407342    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:05:02.410966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:02.410930    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:03.231766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:03.231731    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:03.240501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:03.240451    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:05:03.754317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:03.754286    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:05:03.754825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:03.754452    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:05:04.962138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:04.961937    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:05:04.968289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:04.968263    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:04.982136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:04.976076    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-de4bda\""
Oct 30 19:05:04.982136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:04.976153    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-de4bda-dockercfg-vqp64\""
Oct 30 19:05:04.982136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:04.978872    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:05:05.020582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.020543    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:05:05.025361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.025338    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.083112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.075518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlv8w\" (UniqueName: \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.083112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.075772    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.195137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.186015    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.195137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.186157    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwbx\" (UniqueName: \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.195137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.186250    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlv8w\" (UniqueName: \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.195137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.186419    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.195137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.187414    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.272166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.267198    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:05:05.293112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.291722    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.293112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.291841    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwbx\" (UniqueName: \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.335111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.302032    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.389333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.389304    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwbx\" (UniqueName: \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\") pod \"payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.389515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.389344    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlv8w\" (UniqueName: \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\") pod \"payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.595158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.591848    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:05.666057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.665693    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:05.939053 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:05:05.939024    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8969b3a8_67af_408b_a1a5_d3657496c3d7.slice/crio-e12feca8ecfcd059636c8ad3de38b190febdfc8c19f600c9202183cf37870541 WatchSource:0}: Error finding container e12feca8ecfcd059636c8ad3de38b190febdfc8c19f600c9202183cf37870541: Status 404 returned error can't find the container with id e12feca8ecfcd059636c8ad3de38b190febdfc8c19f600c9202183cf37870541
Oct 30 19:05:05.940528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.940501    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:05:05.994139 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:05:05.994113    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41130fe_bf8b_4ac8_8fd2_75bd98b68832.slice/crio-6a80257a932a693e24a3823d89f5a4487c309deb0661b645cf34e42993ba2d9a WatchSource:0}: Error finding container 6a80257a932a693e24a3823d89f5a4487c309deb0661b645cf34e42993ba2d9a: Status 404 returned error can't find the container with id 6a80257a932a693e24a3823d89f5a4487c309deb0661b645cf34e42993ba2d9a
Oct 30 19:05:05.994500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:05.994201    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:05:06.375611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.375580    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" event={"ID":"8969b3a8-67af-408b-a1a5-d3657496c3d7","Type":"ContainerStarted","Data":"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"}
Oct 30 19:05:06.375795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.375637    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" event={"ID":"8969b3a8-67af-408b-a1a5-d3657496c3d7","Type":"ContainerStarted","Data":"e12feca8ecfcd059636c8ad3de38b190febdfc8c19f600c9202183cf37870541"}
Oct 30 19:05:06.375795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.375671    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:06.377416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.377390    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" event={"ID":"f41130fe-bf8b-4ac8-8fd2-75bd98b68832","Type":"ContainerStarted","Data":"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"}
Oct 30 19:05:06.377553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.377438    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" event={"ID":"f41130fe-bf8b-4ac8-8fd2-75bd98b68832","Type":"ContainerStarted","Data":"6a80257a932a693e24a3823d89f5a4487c309deb0661b645cf34e42993ba2d9a"}
Oct 30 19:05:06.377553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.377470    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:06.413545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.413482    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" podStartSLOduration=2.4134643000000002 podStartE2EDuration="2.4134643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:06.39682763 +0000 UTC m=+1926.660292598" watchObservedRunningTime="2026-10-30 19:05:06.4134643 +0000 UTC m=+1926.676929251"
Oct 30 19:05:06.432352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.432299    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" podStartSLOduration=2.432288645 podStartE2EDuration="2.432288645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:06.430391875 +0000 UTC m=+1926.693856868" watchObservedRunningTime="2026-10-30 19:05:06.432288645 +0000 UTC m=+1926.695753596"
Oct 30 19:05:06.514821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.514788    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:05:06.520316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.520283    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.554426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.554399    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d72beb-openshift-default-dockercfg-drvvh\""
Oct 30 19:05:06.560816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.560786    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:05:06.620103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.619681    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.619841    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.619957    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620003    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620037    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620127    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620159    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620190    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.620410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.620222    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fjt\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.726700    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.726752    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.726784    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.726817    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fjt\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.726952    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.727117    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.727249    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.727302    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.727336    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.728211    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.755131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.743646    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.755131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.743926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.755131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.744471    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.755131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.748396    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.776121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.774681    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fjt\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.782944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.782548    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.793107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.792223    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.793107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.792498    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:06.846386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:06.846078    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:07.301018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:07.300983    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:05:07.301429 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:05:07.301182    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7bb8ad_aa79_430e_91e5_97fd8829c2e4.slice/crio-cbbbc8e768033cc6d4b4c6a314683d694d8190549a5b458d84c9c5779ae0d3c2 WatchSource:0}: Error finding container cbbbc8e768033cc6d4b4c6a314683d694d8190549a5b458d84c9c5779ae0d3c2: Status 404 returned error can't find the container with id cbbbc8e768033cc6d4b4c6a314683d694d8190549a5b458d84c9c5779ae0d3c2
Oct 30 19:05:07.305188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:07.305148    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:07.305282 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:07.305237    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:07.305320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:07.305279    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:07.405963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:07.405926    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerStarted","Data":"cbbbc8e768033cc6d4b4c6a314683d694d8190549a5b458d84c9c5779ae0d3c2"}
Oct 30 19:05:08.474124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:08.473802    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerStarted","Data":"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"}
Oct 30 19:05:08.509154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:08.509074    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podStartSLOduration=2.509056299 podStartE2EDuration="2.509056299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:08.500882589 +0000 UTC m=+1928.764347556" watchObservedRunningTime="2026-10-30 19:05:08.509056299 +0000 UTC m=+1928.772521249"
Oct 30 19:05:08.847736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:08.847702    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:08.853444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:08.853417    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:09.521143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:09.516286    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:09.532533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:09.532501    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:05:10.106113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:10.099333    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:05:10.145704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:10.145644    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:05:11.073760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.073729    5065 scope.go:122] "RemoveContainer" containerID="818d86b9dc67e8d2d2ebbae4199250d105d492eda863281649a08d5ea8752dd7"
Oct 30 19:05:11.088139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.088079    5065 scope.go:122] "RemoveContainer" containerID="cab3446512f30f490233b32e7bc9bf07c148c1f21f858ac511bfc14ff7b4303e"
Oct 30 19:05:11.098960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.098940    5065 scope.go:122] "RemoveContainer" containerID="2d7cf662dbdb21738ed5e091629bea4e478c286140c18a9d1b7f9760b1fe8860"
Oct 30 19:05:11.109537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.109518    5065 scope.go:122] "RemoveContainer" containerID="401765e2db9be581f75779862a0b84179e258ea105496a40635874b0fcec723a"
Oct 30 19:05:11.119717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.119697    5065 scope.go:122] "RemoveContainer" containerID="af61ff5d397fab2ce055462a9785795c11a16a87d50ed424bf0be2ea50010da4"
Oct 30 19:05:11.130130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.130108    5065 scope.go:122] "RemoveContainer" containerID="a7b4c006e723577416525596b4183c8bfb892ba26b88db776cdf9bfe999b7ec8"
Oct 30 19:05:11.142863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.142842    5065 scope.go:122] "RemoveContainer" containerID="c1bd156b7a08436988f52e04258578aaa973bf51d9df7b52a5e6db4ae8331777"
Oct 30 19:05:11.152442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:11.152419    5065 scope.go:122] "RemoveContainer" containerID="715cb99f245bb5aa4683c6cf75436bb62812a548a20516be6a883f3e0fa2c50b"
Oct 30 19:05:12.279306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.279271    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:05:12.288446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.288419    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.295574 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.295548    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:05:12.300128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.299526    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:05:12.313159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.313132    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.313310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.313187    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjph\" (UniqueName: \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.424115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.416000    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.424475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.424443    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjph\" (UniqueName: \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.451667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.451633    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.712157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.711946    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjph\" (UniqueName: \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\") pod \"payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:12.911415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:12.911378    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:13.231937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:13.231907    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:05:13.232251 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:05:13.232229    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8521b31a_c79c_45aa_84bf_47a618e04f3d.slice/crio-17423a66c3f4425b50010f4a40eb9042b4ba8d0de3ea600fe6707bc5990cf346 WatchSource:0}: Error finding container 17423a66c3f4425b50010f4a40eb9042b4ba8d0de3ea600fe6707bc5990cf346: Status 404 returned error can't find the container with id 17423a66c3f4425b50010f4a40eb9042b4ba8d0de3ea600fe6707bc5990cf346
Oct 30 19:05:13.754664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:13.754630    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" event={"ID":"8521b31a-c79c-45aa-84bf-47a618e04f3d","Type":"ContainerStarted","Data":"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"}
Oct 30 19:05:13.755128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:13.754721    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" event={"ID":"8521b31a-c79c-45aa-84bf-47a618e04f3d","Type":"ContainerStarted","Data":"17423a66c3f4425b50010f4a40eb9042b4ba8d0de3ea600fe6707bc5990cf346"}
Oct 30 19:05:13.755128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:13.754753    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:13.777559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:13.777516    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" podStartSLOduration=1.777504157 podStartE2EDuration="1.777504157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:13.77553874 +0000 UTC m=+1934.039003687" watchObservedRunningTime="2026-10-30 19:05:13.777504157 +0000 UTC m=+1934.040969154"
Oct 30 19:05:17.435111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:17.434258    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:05:17.435111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:17.434361    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:05:23.060224 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:05:23.060188    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:05:24.780189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:24.780163    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:05:29.143034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.142952    5065 generic.go:356] "Generic (PLEG): container finished" podID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerID="43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87" exitCode=0
Oct 30 19:05:29.143472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.143040    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerDied","Data":"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"}
Oct 30 19:05:29.143532 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.143518    5065 scope.go:122] "RemoveContainer" containerID="43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"
Oct 30 19:05:29.145309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.145284    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:29.145394 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.145350    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:29.145394 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.145381    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:29.296677 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:29.296642    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:05:30.155511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:30.155473    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerStarted","Data":"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"}
Oct 30 19:05:30.259808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:30.259769    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:05:30.260749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:30.260726    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:30.260839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:30.260784    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:31.260282 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:31.260247    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:31.260756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:31.260294    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:32.260974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:32.260937    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:32.261583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:32.261000    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:33.260298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:33.260268    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:33.260485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:33.260316    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:34.260222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:34.260192    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:34.260640 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:34.260240    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:35.260785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:35.260749    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:05:35.261330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:35.260808    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:05:36.467109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:36.465793    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:05:36.472109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:36.467903    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:05:36.472109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:36.469668    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:05:51.507108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.489297    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:05:51.641195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.587300    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:05:51.641195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.587481    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.738112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.691201    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c6027-openshift-default-dockercfg-7c5ll\""
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.861689    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78k9z\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.861797    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.861845    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.861952    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.861990    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.862013    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.862133    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.862218    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.862265    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.869139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.866008    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:05:51.945039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.922073    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.963781    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.963893    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.963928    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.963951    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964026    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964137    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964293    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78k9z\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964373    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.965316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.964783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.980017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.979570    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:51.980208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:51.980069    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.077202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.042714    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.078046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.077976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.102134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.078495    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.138130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.135818    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78k9z\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.150656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.149531    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.150656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.150067    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:52.208994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:52.208719    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:53.061943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:53.059373    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:05:53.123478 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:05:53.123425    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce02056_7b6e_4a5c_8cc2_3982eebb3e08.slice/crio-aadc262a8fd1f5cb624f0249841abd1b8f7a521611568c054d00bac5244dde20 WatchSource:0}: Error finding container aadc262a8fd1f5cb624f0249841abd1b8f7a521611568c054d00bac5244dde20: Status 404 returned error can't find the container with id aadc262a8fd1f5cb624f0249841abd1b8f7a521611568c054d00bac5244dde20
Oct 30 19:05:53.162245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:53.161728    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:53.162245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:53.161798    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:53.162245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:53.161829    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:05:53.276124 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:05:53.267678    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:05:54.333545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.333500    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerStarted","Data":"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"}
Oct 30 19:05:54.334131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.334107    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerStarted","Data":"aadc262a8fd1f5cb624f0249841abd1b8f7a521611568c054d00bac5244dde20"}
Oct 30 19:05:54.350430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.350372    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podStartSLOduration=3.350350695 podStartE2EDuration="3.350350695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.306529084 +0000 UTC m=+1974.569994034" watchObservedRunningTime="2026-10-30 19:05:54.350350695 +0000 UTC m=+1974.613815643"
Oct 30 19:05:54.360487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.360302    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:54.377160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.376920    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:05:54.377417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.377390    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:05:54.437518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.437464    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:54.452052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.451986    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:05:54.474142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.459519    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:54.583243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.580646    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:54.699015 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:54.698917    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:55.132452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.119307    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:05:55.245409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.245375    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:55.322880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.322853    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:55.331204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.331176    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:05:55.464172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.463462    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:05:55.464172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.463970    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:05:55.937371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:55.937328    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:56.457281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:56.457228    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:56.474114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:56.470168    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:05:56.568738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:56.568694    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:05:56.941969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:56.939257    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:56.941969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:56.939619    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:05:57.443133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.442121    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:05:57.460552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.460502    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:57.482521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.482489    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:05:57.484554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.483132    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:57.637319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.637270    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:57.655112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.646847    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:05:57.879301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.874192    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:57.977135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:57.940298    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:58.359168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.347512    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:05:58.459511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.455519    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:58.459511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.457020    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:05:58.492421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.473288    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:05:58.727169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.637441    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:05:58.727169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.659422    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:05:58.892360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:58.888139    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:05:59.045110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.042508    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:59.045110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.043906    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:05:59.474907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.474300    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:59.577853 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.577821    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:05:59.816280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.816241    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:59.995191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:05:59.995150    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:00.451757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:00.451727    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:06:04.232200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.224290    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"]
Oct 30 19:06:04.313163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.289240    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:04.313163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.289306    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:04.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.337961    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"]
Oct 30 19:06:04.359110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.338102    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.392200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.373726    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:06:04.392200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.373804    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:06:04.417796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.417634    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.419782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.419648    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.420496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.420473    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.528109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.527154    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.528109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.527375    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.528109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.527420    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.529574 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.529533    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.565434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.548932    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.565434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.552934    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\") pod \"maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:04.804321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:04.803548    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:05.272688 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:05.272625    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a0b61c_2aef_44e2_a7a2_05f850448767.slice/crio-de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a WatchSource:0}: Error finding container de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a: Status 404 returned error can't find the container with id de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a
Oct 30 19:06:05.274734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:05.274703    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"]
Oct 30 19:06:05.654683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:05.654591    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw" event={"ID":"f4a0b61c-2aef-44e2-a7a2-05f850448767","Type":"ContainerStarted","Data":"de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a"}
Oct 30 19:06:06.665588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:06.665533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw" event={"ID":"f4a0b61c-2aef-44e2-a7a2-05f850448767","Type":"ContainerStarted","Data":"1ba8bdfb3a54f3ade255a24460f65af56b586398d488efc178e77c146e666028"}
Oct 30 19:06:06.737033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:06.736653    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw" podStartSLOduration=2.736632375 podStartE2EDuration="2.736632375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:06.735422999 +0000 UTC m=+1986.998887964" watchObservedRunningTime="2026-10-30 19:06:06.736632375 +0000 UTC m=+1987.000097325"
Oct 30 19:06:07.677778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:07.677745    5065 generic.go:356] "Generic (PLEG): container finished" podID="f4a0b61c-2aef-44e2-a7a2-05f850448767" containerID="1ba8bdfb3a54f3ade255a24460f65af56b586398d488efc178e77c146e666028" exitCode=0
Oct 30 19:06:07.678279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:07.677832    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw" event={"ID":"f4a0b61c-2aef-44e2-a7a2-05f850448767","Type":"ContainerDied","Data":"1ba8bdfb3a54f3ade255a24460f65af56b586398d488efc178e77c146e666028"}
Oct 30 19:06:09.995855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:09.808205    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:09.996288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:09.995782    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:10.047960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:10.047924    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t4gbc" podUID="46c4e3cc-8401-41d5-83f8-7ea5632ef388" containerName="postgres" probeResult="failure" output="command timed out"
Oct 30 19:06:10.557226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:10.557188    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:10.557466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:10.557252    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:11.278997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:11.278961    5065 scope.go:122] "RemoveContainer" containerID="729313b5b180ea09b9331c0dc7a2a72e69521222cefbc32eb61a7666e3a56866"
Oct 30 19:06:12.445657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.445620    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:12.470711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.470628    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\") pod \"f4a0b61c-2aef-44e2-a7a2-05f850448767\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") "
Oct 30 19:06:12.470877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.470790    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\") pod \"f4a0b61c-2aef-44e2-a7a2-05f850448767\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") "
Oct 30 19:06:12.470877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.470835    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\") pod \"f4a0b61c-2aef-44e2-a7a2-05f850448767\" (UID: \"f4a0b61c-2aef-44e2-a7a2-05f850448767\") "
Oct 30 19:06:12.472395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.472347    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca" pod "f4a0b61c-2aef-44e2-a7a2-05f850448767" (UID: "f4a0b61c-2aef-44e2-a7a2-05f850448767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:12.573688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.573635    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:12.580014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.579976    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca" pod "f4a0b61c-2aef-44e2-a7a2-05f850448767" (UID: "f4a0b61c-2aef-44e2-a7a2-05f850448767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:12.595669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.595612    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5" pod "f4a0b61c-2aef-44e2-a7a2-05f850448767" (UID: "f4a0b61c-2aef-44e2-a7a2-05f850448767"). InnerVolumeSpecName "kube-api-access-frxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:12.674957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.674926    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/f4a0b61c-2aef-44e2-a7a2-05f850448767-kube-api-access-frxq5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:12.675114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.674961    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a0b61c-2aef-44e2-a7a2-05f850448767-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:12.913702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.913659    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw" event={"ID":"f4a0b61c-2aef-44e2-a7a2-05f850448767","Type":"ContainerDied","Data":"de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a"}
Oct 30 19:06:12.913986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.913742    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de79b5e927243c8c6a6bc785feb1bbcbcb85206c519a9e7ea362754b4a30875a"
Oct 30 19:06:12.913986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:12.913784    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"
Oct 30 19:06:14.369855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:14.369821    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:06:14.390260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:14.390210    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:06:15.220311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.218358    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:06:15.226261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.226163    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f4a0b61c-2aef-44e2-a7a2-05f850448767" containerName="revoke-keys"
Oct 30 19:06:15.226261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.226193    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a0b61c-2aef-44e2-a7a2-05f850448767" containerName="revoke-keys"
Oct 30 19:06:15.227351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.226736    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a0b61c-2aef-44e2-a7a2-05f850448767" containerName="revoke-keys"
Oct 30 19:06:15.415374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.413327    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:06:15.415374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.413444    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.480525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480438    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrj9n\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.480693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480611    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.480693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480657    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480700    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480748    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.480925    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.481060    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.481182    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.481305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.481219    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583325    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583404    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583549    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrj9n\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583604    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583635    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583682    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583719    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583807    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583881    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.583916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.583895    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.584238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.584218    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.585167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.584866    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.585167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.584920    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.585167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.585000    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.646755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.646715    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.646924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.646810    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.646924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.646913    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrj9n\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.649621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.649591    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:15.654830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:15.654805    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:16.484513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.484481    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:06:16.487653 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:16.487604    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcc55db_906b_464c_98d4_18c6a28f32a0.slice/crio-87a8e1ccc0cd648369ff4cb9ba1b4106d360b26894bc2354e3630fa4b3e090e3 WatchSource:0}: Error finding container 87a8e1ccc0cd648369ff4cb9ba1b4106d360b26894bc2354e3630fa4b3e090e3: Status 404 returned error can't find the container with id 87a8e1ccc0cd648369ff4cb9ba1b4106d360b26894bc2354e3630fa4b3e090e3
Oct 30 19:06:16.493447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.493409    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:16.493566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.493531    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:16.493629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.493578    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:16.971925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.971348    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" event={"ID":"6dcc55db-906b-464c-98d4-18c6a28f32a0","Type":"ContainerStarted","Data":"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"}
Oct 30 19:06:16.971925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:16.971412    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" event={"ID":"6dcc55db-906b-464c-98d4-18c6a28f32a0","Type":"ContainerStarted","Data":"87a8e1ccc0cd648369ff4cb9ba1b4106d360b26894bc2354e3630fa4b3e090e3"}
Oct 30 19:06:17.034368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:17.034300    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podStartSLOduration=2.03428046 podStartE2EDuration="2.03428046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:17.02789022 +0000 UTC m=+1997.291355176" watchObservedRunningTime="2026-10-30 19:06:17.03428046 +0000 UTC m=+1997.297745414"
Oct 30 19:06:17.673654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:17.669359    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:17.673654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:17.669481    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:17.673654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:17.669526    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:18.362246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:18.355111    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": read tcp 10.132.0.2:33770->10.132.1.26:15021: read: connection reset by peer" start-of-body=
Oct 30 19:06:18.362246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:18.355185    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": read tcp 10.132.0.2:33770->10.132.1.26:15021: read: connection reset by peer"
Oct 30 19:06:18.656026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:18.655940    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:18.656026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:18.655996    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:19.024017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.022503    5065 generic.go:356] "Generic (PLEG): container finished" podID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerID="fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2" exitCode=0
Oct 30 19:06:19.024017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.023766    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerDied","Data":"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"}
Oct 30 19:06:19.024645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.024235    5065 scope.go:122] "RemoveContainer" containerID="fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"
Oct 30 19:06:19.032467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.032425    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:19.032624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.032534    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:19.032696 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.032622    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:19.388265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.379612    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerName="payload-processing" containerID="cri-o://ea7373657459d5482d6216f9b97434138a1ad0a204f332d016a4b013ee6e1f35" gracePeriod=25
Oct 30 19:06:19.420139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.417352    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerName="payload-pre-processing" containerID="cri-o://319aca329c3490400954bd18402ee048972ddb9acb189949da6ff4de74f28cf4" gracePeriod=25
Oct 30 19:06:19.655310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.655215    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:19.655310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:19.655283    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:20.094562 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:20.093884    5065 logging.go:55] [core] [Channel #4448 SubChannel #4450]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.25:50052", ServerName: "10.132.1.25:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.25:50052: connect: connection refused"
Oct 30 19:06:20.103978 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:20.102223    5065 logging.go:55] [core] [Channel #4449 SubChannel #4451]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.24:50052", ServerName: "10.132.1.24:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.24:50052: connect: connection refused"
Oct 30 19:06:20.149117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.146979    5065 generic.go:356] "Generic (PLEG): container finished" podID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerID="ea7373657459d5482d6216f9b97434138a1ad0a204f332d016a4b013ee6e1f35" exitCode=0
Oct 30 19:06:20.149117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.147213    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" event={"ID":"294bf429-04b4-414b-9c6f-9dd466ea723b","Type":"ContainerDied","Data":"ea7373657459d5482d6216f9b97434138a1ad0a204f332d016a4b013ee6e1f35"}
Oct 30 19:06:20.281140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.279740    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerStarted","Data":"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"}
Oct 30 19:06:20.293754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.293696    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:20.346861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.332519    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.347711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.347343    5065 generic.go:356] "Generic (PLEG): container finished" podID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerID="319aca329c3490400954bd18402ee048972ddb9acb189949da6ff4de74f28cf4" exitCode=0
Oct 30 19:06:20.347711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.347564    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" event={"ID":"c76d05b0-f43a-480b-8d7c-627f6d37d8ad","Type":"ContainerDied","Data":"319aca329c3490400954bd18402ee048972ddb9acb189949da6ff4de74f28cf4"}
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.385792    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.385912    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386014    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386064    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386183    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386256    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386306    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.386349    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h2w\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.399106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.390715    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:20.420979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.405518    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:06:20.420979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.405624    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:20.420979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.405665    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:20.420979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.419562    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.529840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.529785    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.529863    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.529907    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h2w\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530017    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530175    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530213    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530295    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530349    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.530454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.530425    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.531766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.531694    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.532302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.532238    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.532539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.532480    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.537708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.537666    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.537825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.537714    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.538645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.538590    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.544517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.544474    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.547657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.547618    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h2w\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.549049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.549016    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:20.565368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.565250    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:06:20.647889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.646943    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\" (UniqueName: \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\") pod \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") "
Oct 30 19:06:20.647889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.647222    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\") pod \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\" (UID: \"c76d05b0-f43a-480b-8d7c-627f6d37d8ad\") "
Oct 30 19:06:20.654555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.650266    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume" pod "c76d05b0-f43a-480b-8d7c-627f6d37d8ad" (UID: "c76d05b0-f43a-480b-8d7c-627f6d37d8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:20.690422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.668238    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:20.690422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.668301    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:20.750107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.732728    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22" pod "c76d05b0-f43a-480b-8d7c-627f6d37d8ad" (UID: "c76d05b0-f43a-480b-8d7c-627f6d37d8ad"). InnerVolumeSpecName "kube-api-access-f8s22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:20.752307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.752245    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8s22\" (UniqueName: \"kubernetes.io/projected/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-kube-api-access-f8s22\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:20.752307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.752281    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d05b0-f43a-480b-8d7c-627f6d37d8ad-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:20.775831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:20.775776    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:21.068945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.068906    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:06:21.089700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.089311    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.24:50052\" within 1s: context deadline exceeded"
Oct 30 19:06:21.089700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.089446    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.25:50052\" within 1s: context deadline exceeded"
Oct 30 19:06:21.194607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.192945    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\") pod \"294bf429-04b4-414b-9c6f-9dd466ea723b\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") "
Oct 30 19:06:21.194607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.193131    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\" (UniqueName: \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\") pod \"294bf429-04b4-414b-9c6f-9dd466ea723b\" (UID: \"294bf429-04b4-414b-9c6f-9dd466ea723b\") "
Oct 30 19:06:21.218566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.195402    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume" pod "294bf429-04b4-414b-9c6f-9dd466ea723b" (UID: "294bf429-04b4-414b-9c6f-9dd466ea723b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:21.253111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.252339    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj" pod "294bf429-04b4-414b-9c6f-9dd466ea723b" (UID: "294bf429-04b4-414b-9c6f-9dd466ea723b"). InnerVolumeSpecName "kube-api-access-v57xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:21.326114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.321591    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/294bf429-04b4-414b-9c6f-9dd466ea723b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:21.326114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.321635    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57xj\" (UniqueName: \"kubernetes.io/projected/294bf429-04b4-414b-9c6f-9dd466ea723b-kube-api-access-v57xj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:21.364713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.364681    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"
Oct 30 19:06:21.365275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.365250    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w" event={"ID":"294bf429-04b4-414b-9c6f-9dd466ea723b","Type":"ContainerDied","Data":"9b57189cf089263f322f1402c1d6a053d963c36ff332f9d07e14f76e3441f3c9"}
Oct 30 19:06:21.365420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.365335    5065 scope.go:122] "RemoveContainer" containerID="ea7373657459d5482d6216f9b97434138a1ad0a204f332d016a4b013ee6e1f35"
Oct 30 19:06:21.371514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.371482    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"
Oct 30 19:06:21.371663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.371536    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd" event={"ID":"c76d05b0-f43a-480b-8d7c-627f6d37d8ad","Type":"ContainerDied","Data":"86c82fd2266f442802707015fde58eae493ce19b2abb097670c90ea3a01a7698"}
Oct 30 19:06:21.440290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.433653    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:21.440290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.433713    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:21.461256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.461214    5065 scope.go:122] "RemoveContainer" containerID="319aca329c3490400954bd18402ee048972ddb9acb189949da6ff4de74f28cf4"
Oct 30 19:06:21.467037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.467015    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:06:21.475162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.475056    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c9e9de-bb6c85fcf-k6pdd"]
Oct 30 19:06:21.525827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.525792    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:06:21.563280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.563259    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c9e9de-8dcdcd85d-69z4w"]
Oct 30 19:06:21.565636 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:21.565605    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bacc79_1c1d_42ab_bd2c_924e2ae87131.slice/crio-47b013119fa3c2ce8b65ab420655c6720656960a9392639c9e8ac750b59cb10c WatchSource:0}: Error finding container 47b013119fa3c2ce8b65ab420655c6720656960a9392639c9e8ac750b59cb10c: Status 404 returned error can't find the container with id 47b013119fa3c2ce8b65ab420655c6720656960a9392639c9e8ac750b59cb10c
Oct 30 19:06:21.569615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.569581    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:21.569807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.569788    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:21.569966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.569945    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:21.571080 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.571056    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:21.663109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.656528    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:21.663109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:21.656600    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:22.157832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.157797    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" path="/var/lib/kubelet/pods/294bf429-04b4-414b-9c6f-9dd466ea723b/volumes"
Oct 30 19:06:22.159218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.159186    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" path="/var/lib/kubelet/pods/c76d05b0-f43a-480b-8d7c-627f6d37d8ad/volumes"
Oct 30 19:06:22.159709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.159689    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:06:22.161150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161127    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerName="payload-pre-processing"
Oct 30 19:06:22.161238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161151    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerName="payload-pre-processing"
Oct 30 19:06:22.161238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161196    5065 cpu_manager.go:414] "containerMap: removing container" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerName="payload-processing"
Oct 30 19:06:22.161238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161205    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerName="payload-processing"
Oct 30 19:06:22.161399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161387    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76d05b0-f43a-480b-8d7c-627f6d37d8ad" containerName="payload-pre-processing"
Oct 30 19:06:22.161439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.161409    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="294bf429-04b4-414b-9c6f-9dd466ea723b" containerName="payload-processing"
Oct 30 19:06:22.176472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.176440    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:06:22.176563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.176515    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.348491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.348395    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.349343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.349163    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.349633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.349560    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.349882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.349863    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.350507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.350487    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.350691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.350674    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfsk\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.350878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.350861    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.351119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.351078    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.351342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.351326    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.542681    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.542771    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.542872    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.542902    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfsk\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.542931    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.548119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.543019    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.563237    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.563572    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.563655    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.564205    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.564474    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.574825    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.574886    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.575715    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.575982    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.577000    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.631161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.578790    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.670232    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.670289    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.671765    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" event={"ID":"04bacc79-1c1d-42ab-bd2c-924e2ae87131","Type":"ContainerStarted","Data":"aa04c0d7e90c904b42b6e0767bf2a7b0215e1cb2472ab2212a688cd5b31bc048"}
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.672326    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" event={"ID":"04bacc79-1c1d-42ab-bd2c-924e2ae87131","Type":"ContainerStarted","Data":"47b013119fa3c2ce8b65ab420655c6720656960a9392639c9e8ac750b59cb10c"}
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.675187    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.678539    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.692114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.691504    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfsk\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:22.747698 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.745742    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podStartSLOduration=2.745706647 podStartE2EDuration="2.745706647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.716527395 +0000 UTC m=+2002.979992375" watchObservedRunningTime="2026-10-30 19:06:22.745706647 +0000 UTC m=+2003.009171597"
Oct 30 19:06:22.833235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.833182    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:22.921360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:22.921279    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:23.407003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.406482    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:23.407003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.406542    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:23.656415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.656379    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 19:06:23.656415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.656402    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:06:23.656697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.656434    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 19:06:23.677133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.676934    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:23.677133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.677017    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:23.677133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.677060    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:23.761113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.749978    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" event={"ID":"ba32d319-54ac-4975-bf83-35d51f752fd1","Type":"ContainerStarted","Data":"650a079532573dde66c735af7fe7c4413d686f1964f3397f4a03907894bf6fa4"}
Oct 30 19:06:23.774599 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:06:23.774464    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:06:23.778624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.778597    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:23.779063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:23.779038    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:24.410113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.409250    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:24.410113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.409311    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:24.663789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.663675    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:24.665048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.664923    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:24.666010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.665959    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:06:24.805717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.805293    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:24.805717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.805345    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:24.824729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.819644    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:24.852038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:24.825010    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:25.010870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.010836    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" event={"ID":"ba32d319-54ac-4975-bf83-35d51f752fd1","Type":"ContainerStarted","Data":"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"}
Oct 30 19:06:25.454922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.454821    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.455588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.454910    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:25.560944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.560911    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:25.562212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.561256    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:25.776589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.776559    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.776839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.776803    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:25.957598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.957456    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.957598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.957514    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:25.965549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:25.965527    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:26.455597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:26.455554    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:06:26.456692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:26.456652    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:06:26.790148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:26.776925    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:26.790148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:26.776970    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:27.081159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.065765    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:27.081159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.065820    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:27.748703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.735894    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podStartSLOduration=6.735872537 podStartE2EDuration="6.735872537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:25.144538866 +0000 UTC m=+2005.408003826" watchObservedRunningTime="2026-10-30 19:06:27.735872537 +0000 UTC m=+2007.999337486"
Oct 30 19:06:27.762126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.749691    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:06:27.762126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.754931    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:06:27.784125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.775434    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:06:27.799158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.785502    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:27.799158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.785556    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:27.857158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:27.846602    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"]
Oct 30 19:06:28.078213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.053239    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:28.078213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.053295    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:28.251074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.251046    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c9e9de-tdzlw"]
Oct 30 19:06:28.788758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.778470    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:28.788758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.778530    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:28.981111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.973613    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:28.981111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:28.973671    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:29.776412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:29.776377    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 19:06:29.843147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:29.776448    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 19:06:29.970910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:29.970879    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:29.971326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:29.971273    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:29.974346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:29.974319    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a0b61c-2aef-44e2-a7a2-05f850448767" path="/var/lib/kubelet/pods/f4a0b61c-2aef-44e2-a7a2-05f850448767/volumes"
Oct 30 19:06:30.356483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.356389    5065 generic.go:356] "Generic (PLEG): container finished" podID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerID="ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73" exitCode=0
Oct 30 19:06:30.356703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.356671    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerDied","Data":"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"}
Oct 30 19:06:30.357244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.357199    5065 scope.go:122] "RemoveContainer" containerID="ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"
Oct 30 19:06:30.359399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.359359    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.359504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.359460    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.359546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.359511    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.360397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.359957    5065 generic.go:356] "Generic (PLEG): container finished" podID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerID="b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9" exitCode=0
Oct 30 19:06:30.360397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.360000    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerDied","Data":"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"}
Oct 30 19:06:30.369203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.360430    5065 scope.go:122] "RemoveContainer" containerID="b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"
Oct 30 19:06:30.441112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.441041    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.441328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.441223    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.441328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.441281    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:30.887161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.886410    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:30.891124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.887886    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:30.937131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.936781    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:30.957591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.957168    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:30.957591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:30.957222    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:31.372920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.372878    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerStarted","Data":"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"}
Oct 30 19:06:31.377336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.376559    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerStarted","Data":"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"}
Oct 30 19:06:31.865342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.850807    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:06:31.872444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.872412    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:31.880706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.880670    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:31.956327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.953518    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:31.956327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:31.953571    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:32.069777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.069745    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:06:32.070268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.070233    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:32.070411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.070305    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:32.859114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.854641    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:32.859114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.854699    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:32.971132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.959377    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Oct 30 19:06:32.971132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:32.959435    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Oct 30 19:06:33.090122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:33.086259    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:33.090122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:33.086328    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:33.853555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:33.853514    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:33.853785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:33.853581    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:34.067113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.066434    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:34.067113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.066494    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:34.291750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.291718    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:34.294061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.294041    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:34.300537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.300512    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:06:34.847190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.847157    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:34.847469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:34.847442    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:35.072674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.072620    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:35.073362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.073262    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:35.096128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.093624    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:06:35.201801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.200295    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:06:35.201801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.200424    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.224125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.218501    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"routing-overlay\""
Oct 30 19:06:35.224125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.218601    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"openshift-service-ca.crt\""
Oct 30 19:06:35.231121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.227321    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-e2e--0d1ddd5a-dockercfg-7twb9\""
Oct 30 19:06:35.231121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.227414    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"kube-root-ca.crt\""
Oct 30 19:06:35.314593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.313278    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-394aab6f23b5\""
Oct 30 19:06:35.314593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.313389    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e94aa\""
Oct 30 19:06:35.410226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.409474    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.410226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.409660    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.410226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.409965    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdqw\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.410226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.410007    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.521802 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.521756    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdqw\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.522529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.522503    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.522896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.522877    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.523114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.523069    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.528777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.528753    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.549059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.549033    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.554618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.554594    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdqw\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.588120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.585162    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.687108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.687062    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:35.848697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.848361    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:35.848697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:35.848408    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:36.088166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.071579    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:36.088166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.071658    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:36.473257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.473211    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:06:36.477009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.476972    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:36.722168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.719191    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 19:06:36.874661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.874621    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:36.874967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:36.874936    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:37.064891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.064855    5065 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 30 19:06:37.065232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.065199    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 30 19:06:37.158159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.154187    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:06:37.158159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.154436    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy" containerID="cri-o://77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2" gracePeriod=30
Oct 30 19:06:37.262777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.262741    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:37.263242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.263214    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy" containerID="cri-o://aa04c0d7e90c904b42b6e0767bf2a7b0215e1cb2472ab2212a688cd5b31bc048" gracePeriod=30
Oct 30 19:06:37.338618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.276807    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:06:37.853276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.853238    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:37.853959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:37.853925    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:38.288995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.288959    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" event={"ID":"c937ac0b-8811-4772-b54e-6bd80e522c57","Type":"ContainerStarted","Data":"21ec09190a0cc6c5d2759f9475ca19f83cf601a38a92264ed18595ce05c5d1db"}
Oct 30 19:06:38.294188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.294154    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" event={"ID":"c937ac0b-8811-4772-b54e-6bd80e522c57","Type":"ContainerStarted","Data":"2a65deeec9939b37ad07a8153c2c67b3e1424b05e93dfcc2259c4dc3376295fb"}
Oct 30 19:06:38.298114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.294588    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:38.319239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.319183    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" podStartSLOduration=4.319165446 podStartE2EDuration="4.319165446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:38.318629898 +0000 UTC m=+2018.582094846" watchObservedRunningTime="2026-10-30 19:06:38.319165446 +0000 UTC m=+2018.582630394"
Oct 30 19:06:38.750129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.749973    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:06:38.848995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.848378    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:38.848995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:38.848433    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:39.058403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.058361    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:06:39.096822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.083277    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:06:39.096822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.083402    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.118731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.117876    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-689c23-openshift-default-dockercfg-ng4kn\""
Oct 30 19:06:39.465589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.465500    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.468533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.467534    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.487539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.487506    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.488488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.488464    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.488750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.488731    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.488949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.488931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd7l\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.489199 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.489182    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.489640 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.489624    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.489999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.489982    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.516855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.516816    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:06:39.594205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.594161    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.595131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.595103    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd7l\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.595924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.595905    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.603266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.603244    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.608186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.608167    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.608681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.608664    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.609367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.609345    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.667974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.667940    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.668616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.668594    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.675903 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.609737    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.623569    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.649599    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd7l\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.608582    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.668487    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.607452    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.676743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.596760    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.697763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.687788    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.698006 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.690355    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:39.850511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.850461    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:06:39.850839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:39.850788    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:06:40.002438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.002404    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:40.413040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.413011    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:06:40.425116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.424470    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:40.425116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.424563    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:40.425116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.424607    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:06:40.507173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:40.507070    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerStarted","Data":"c1d64568d9f6a1066f36d0cccd92144f8543c88cba89ec16489f6b04781611e5"}
Oct 30 19:06:41.366259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:41.366229    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:06:41.387163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:41.384647    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:06:41.544428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:41.544388    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerStarted","Data":"082c5628cc7d227e9422e9575248af0d2d88768c27467a923d3336c4d794410f"}
Oct 30 19:06:41.617784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:41.617672    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podStartSLOduration=3.61765326 podStartE2EDuration="3.61765326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:41.617151993 +0000 UTC m=+2021.880616954" watchObservedRunningTime="2026-10-30 19:06:41.61765326 +0000 UTC m=+2021.881118210"
Oct 30 19:06:42.003054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.002961    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:42.010610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.010577    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:42.553601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.553566    5065 generic.go:356] "Generic (PLEG): container finished" podID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerID="aa04c0d7e90c904b42b6e0767bf2a7b0215e1cb2472ab2212a688cd5b31bc048" exitCode=0
Oct 30 19:06:42.554051 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.553633    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" event={"ID":"04bacc79-1c1d-42ab-bd2c-924e2ae87131","Type":"ContainerDied","Data":"aa04c0d7e90c904b42b6e0767bf2a7b0215e1cb2472ab2212a688cd5b31bc048"}
Oct 30 19:06:42.554224 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.554111    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:42.555327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.555309    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:06:42.599066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.599038    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634233    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634305    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634340    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634394    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634467    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634607    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634654    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634771    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.636250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.634829    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\") pod \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\" (UID: \"04bacc79-1c1d-42ab-bd2c-924e2ae87131\") "
Oct 30 19:06:42.637864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.637245    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.638147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.638125    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.638882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.638840    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.639032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.638923    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.640333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.640304    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:42.643218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.643188    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.643834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.643803    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.645682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.645648    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "kube-api-access-w4h2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.649033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.648998    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo" pod "04bacc79-1c1d-42ab-bd2c-924e2ae87131" (UID: "04bacc79-1c1d-42ab-bd2c-924e2ae87131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750659    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750698    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750716    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750731    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750745    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750761    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750775    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750790    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04bacc79-1c1d-42ab-bd2c-924e2ae87131-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.750980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:42.750803    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4h2w\" (UniqueName: \"kubernetes.io/projected/04bacc79-1c1d-42ab-bd2c-924e2ae87131-kube-api-access-w4h2w\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:43.621043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.620345    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"
Oct 30 19:06:43.624470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.624415    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc" event={"ID":"04bacc79-1c1d-42ab-bd2c-924e2ae87131","Type":"ContainerDied","Data":"47b013119fa3c2ce8b65ab420655c6720656960a9392639c9e8ac750b59cb10c"}
Oct 30 19:06:43.624598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.624536    5065 scope.go:122] "RemoveContainer" containerID="aa04c0d7e90c904b42b6e0767bf2a7b0215e1cb2472ab2212a688cd5b31bc048"
Oct 30 19:06:43.661174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.661130    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:43.668206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.668180    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-j7npc"]
Oct 30 19:06:43.893572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:43.893493    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" path="/var/lib/kubelet/pods/04bacc79-1c1d-42ab-bd2c-924e2ae87131/volumes"
Oct 30 19:06:46.708459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.708430    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:06:46.730911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.730878    5065 generic.go:356] "Generic (PLEG): container finished" podID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerID="77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2" exitCode=0
Oct 30 19:06:46.731061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.730934    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"
Oct 30 19:06:46.731061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.730942    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerDied","Data":"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"}
Oct 30 19:06:46.731061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.731000    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs" event={"ID":"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f","Type":"ContainerDied","Data":"ff5bd4a5456fed889677290fe0938f7e05a877773fd6bfc48cc9e005dc948eca"}
Oct 30 19:06:46.731061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.731031    5065 scope.go:122] "RemoveContainer" containerID="77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"
Oct 30 19:06:46.742524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.742515    5065 scope.go:122] "RemoveContainer" containerID="ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"
Oct 30 19:06:46.756581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.756555    5065 scope.go:122] "RemoveContainer" containerID="77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"
Oct 30 19:06:46.756970 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:06:46.756941    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2\": container with ID starting with 77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2 not found: ID does not exist" containerID="77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"
Oct 30 19:06:46.757131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.756979    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2"} err="failed to get container status \"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2\": rpc error: code = NotFound desc = could not find container \"77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2\": container with ID starting with 77a381cdb2f0a2822f0a4fe360cc25eb734f5e47dfc8adaa1748a8c5dde097a2 not found: ID does not exist"
Oct 30 19:06:46.757131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.757002    5065 scope.go:122] "RemoveContainer" containerID="ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"
Oct 30 19:06:46.757394 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:06:46.757366    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73\": container with ID starting with ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73 not found: ID does not exist" containerID="ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"
Oct 30 19:06:46.757517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.757400    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73"} err="failed to get container status \"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73\": rpc error: code = NotFound desc = could not find container \"ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73\": container with ID starting with ca3f4ee24b4e927d7af49b906c162590e8f1bbeabd206b6affa5f349f80ebf73 not found: ID does not exist"
Oct 30 19:06:46.779293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779259    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.779426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779316    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.779533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779514    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.779617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779525    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:46.779617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779604    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779735    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779862    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779898    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779920    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.779989    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.780120    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\") pod \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\" (UID: \"8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f\") "
Oct 30 19:06:46.780344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.780139    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:46.780866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.780415    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:46.781004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.780979    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.781073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.781012    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.781073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.781038    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.781073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.781052    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.781073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.781066    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.783923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.783882    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:46.784391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.784350    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:46.784391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.784361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:46.784674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.784647    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp" pod "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" (UID: "8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f"). InnerVolumeSpecName "kube-api-access-xm2hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:46.883324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.883292    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.883324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.883324    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.883568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.883341    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm2hp\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-kube-api-access-xm2hp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:46.883568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:46.883360    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.168111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:47.164143    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:06:47.192382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:47.186493    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-dpwbs"]
Oct 30 19:06:47.931274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:47.931241    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" path="/var/lib/kubelet/pods/8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f/volumes"
Oct 30 19:06:50.513800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:50.513763    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:06:51.976248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.976209    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:06:51.978048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978019    5065 cpu_manager.go:414] "containerMap: removing container" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy"
Oct 30 19:06:51.978048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978048    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy"
Oct 30 19:06:51.978258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978162    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:06:51.978258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978170    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:06:51.978258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978235    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:06:51.978258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978245    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:06:51.978610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978594    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:06:51.978752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.978626    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bacc79-1c1d-42ab-bd2c-924e2ae87131" containerName="istio-proxy"
Oct 30 19:06:51.988893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.988868    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:51.989146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.989120    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:06:51.996408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.996386    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-9ee505a9f2b0\""
Oct 30 19:06:51.996870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.996426    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"kube-root-ca.crt\""
Oct 30 19:06:51.996870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:51.996436    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"openshift-service-ca.crt\""
Oct 30 19:06:52.035732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.035702    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"routing-overlay\""
Oct 30 19:06:52.036096 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.036054    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c6027\""
Oct 30 19:06:52.088617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.085311    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.088617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.085411    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zv84\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.117173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.107253    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.117173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.107368    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.117173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.109488    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-e2e--a67c8769-dockercfg-pnpzq\""
Oct 30 19:06:52.252187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.244578    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zv84\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.272079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.272049    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.286949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.286916    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.292216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.287763    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.300225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.295513    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.311290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.296972    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.311525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.308477    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.337332 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.337250    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zv84\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.627775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.627741    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:52.922562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:52.922522    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:06:52.922744 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:06:52.922720    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adfcb30_a158_4a1c_8646_a0844b1ff6c0.slice/crio-a6e8e954c86f1fb2dd0d82ed387126eab1a0a905ee77f274dbfae5e8f312407a WatchSource:0}: Error finding container a6e8e954c86f1fb2dd0d82ed387126eab1a0a905ee77f274dbfae5e8f312407a: Status 404 returned error can't find the container with id a6e8e954c86f1fb2dd0d82ed387126eab1a0a905ee77f274dbfae5e8f312407a
Oct 30 19:06:53.058922 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:06:53.058883    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:06:53.823360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:53.823316    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" event={"ID":"4adfcb30-a158-4a1c-8646-a0844b1ff6c0","Type":"ContainerStarted","Data":"29cd0cb60da2cb2765b2b5a19084cbcb1486bad3d79c8ed99bb7e502e69e70a8"}
Oct 30 19:06:53.823552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:53.823384    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:06:53.823552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:53.823404    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" event={"ID":"4adfcb30-a158-4a1c-8646-a0844b1ff6c0","Type":"ContainerStarted","Data":"a6e8e954c86f1fb2dd0d82ed387126eab1a0a905ee77f274dbfae5e8f312407a"}
Oct 30 19:06:53.876008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:06:53.875959    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" podStartSLOduration=2.875946663 podStartE2EDuration="2.875946663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:53.872317766 +0000 UTC m=+2034.135782733" watchObservedRunningTime="2026-10-30 19:06:53.875946663 +0000 UTC m=+2034.139411615"
Oct 30 19:07:00.329129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.329055    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:07:00.330784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.330757    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e83d2a2-f6fa-4a6b-bd78-1c947eaf1a5f" containerName="istio-proxy"
Oct 30 19:07:00.337870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.337847    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.366229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.361494    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:07:00.428544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428498    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.428821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428596    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.428821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428668    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.428821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428767    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.429034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428894    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.429034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428945    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.429034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.428993    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.429034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.429023    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.430466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.429065    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7rt\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.530672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.530632    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.530863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.530732    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.530863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.530802    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.530996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.530896    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.530996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.530967    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531037    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531073    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531136    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531507    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531541    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7rt\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531406    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531564    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531326    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.531480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.531812    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.534018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.533992    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.534174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.534001    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.548816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.548732    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7rt\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.551453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.551427    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:00.663760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:00.663675    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:01.074899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:01.074870    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:07:01.077661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:01.077617    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:01.077758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:01.077715    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:01.077803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:01.077787    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:02.155890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.155842    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" event={"ID":"f99cf479-590c-47e7-a6de-e1be93a0a0df","Type":"ContainerStarted","Data":"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"}
Oct 30 19:07:02.156412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.155917    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" event={"ID":"f99cf479-590c-47e7-a6de-e1be93a0a0df","Type":"ContainerStarted","Data":"9449062b26b7b2f958427d3c728e85f8b21ad755f3c9669e451ff98c053df534"}
Oct 30 19:07:02.246252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.246193    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podStartSLOduration=2.246175882 podStartE2EDuration="2.246175882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:02.202565377 +0000 UTC m=+2042.466030327" watchObservedRunningTime="2026-10-30 19:07:02.246175882 +0000 UTC m=+2042.509640829"
Oct 30 19:07:02.664178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.664146    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:02.665736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.665703    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:02.665866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:02.665753    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:03.664853 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:03.664823    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:03.665286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:03.664884    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:04.664384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:04.664345    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:04.664580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:04.664412    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:04.850268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:04.850235    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:07:05.664682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:05.664649    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:05.664933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:05.664701    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:06.664205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:06.664174    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:06.664687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:06.664234    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:06.970713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:06.970635    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:07:07.036378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.031669    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.043187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.043157    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:07:07.078988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.078963    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.111248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.111212    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.111515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.111494    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.111716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.111686    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cmw6\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.111949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.111931    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.112233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.112210    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.112402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.112385    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.112641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.112620    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.112855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.112837    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214068    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214196    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214269    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214287    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214316    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214353    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cmw6\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214419    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214458    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.214567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.214478    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.215184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.215156    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.215400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.215382    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.217007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.216957    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.217335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.217307    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.217519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.217393    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.218310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.218293    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.218571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.218519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.233514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.233447    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.233657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.233634    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cmw6\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.353655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.353623    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:07.664947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.664913    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 19:07:07.665589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.664974    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 19:07:07.717740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.717175    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:07.717740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.717264    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:07.717740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.717309    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:07.731886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:07.731854    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:07:08.331981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.331939    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" event={"ID":"5dbafd20-cb17-4156-ac4f-360d7649461f","Type":"ContainerStarted","Data":"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"}
Oct 30 19:07:08.332228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.332007    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" event={"ID":"5dbafd20-cb17-4156-ac4f-360d7649461f","Type":"ContainerStarted","Data":"512be5fcb5d17e96e25f55100531679d1197ab9811ccc574fbd52924e38227df"}
Oct 30 19:07:08.359305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.359271    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:08.396110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.391693    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podStartSLOduration=2.391672657 podStartE2EDuration="2.391672657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:08.360565463 +0000 UTC m=+2048.624030414" watchObservedRunningTime="2026-10-30 19:07:08.391672657 +0000 UTC m=+2048.655137606"
Oct 30 19:07:08.716359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.715586    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:08.720872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.717257    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:08.729940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:08.729401    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:07:09.356026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:09.355998    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:09.356237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:09.356047    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:10.358111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:10.357478    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:10.358111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:10.357539    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:11.354273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:11.354243    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:11.354588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:11.354295    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:11.374218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:11.374183    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:07:11.374580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:11.374229    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:07:11.910109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:11.892107    5065 scope.go:122] "RemoveContainer" containerID="2dd7b56022fbbed7b3b9b2a977d98e1d9a6f17a35d9ccf1a5530899175d8c441"
Oct 30 19:07:12.358111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:12.354559    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:12.358111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:12.354636    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:13.354287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:13.354256    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:13.354694 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:13.354310    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:14.359606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:14.354744    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 19:07:14.359606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:14.354800    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 19:07:15.360438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:15.360399    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:15.361574 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:15.361551    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:15.362305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:15.362283    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:07:19.627438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.627407    5065 generic.go:356] "Generic (PLEG): container finished" podID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerID="2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b" exitCode=0
Oct 30 19:07:19.627906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.627488    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerDied","Data":"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"}
Oct 30 19:07:19.627983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.627967    5065 scope.go:122] "RemoveContainer" containerID="2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"
Oct 30 19:07:19.629791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.629761    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:19.629858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.629829    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:19.629896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:19.629860    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:20.639664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:20.639632    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerStarted","Data":"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"}
Oct 30 19:07:21.210375 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:21.210341    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:07:21.211059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:21.211040    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:07:21.211147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:21.211103    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:07:22.210638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:22.210599    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:07:22.211204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:22.210673    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:07:23.061662 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:07:23.061628    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:07:23.211488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:23.211445    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:07:23.212029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:23.211509    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:07:24.210484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:24.210443    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:07:24.210668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:24.210503    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:07:25.210508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:25.210472    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 19:07:25.211022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:25.210537    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 19:07:25.323316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:25.323279    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:07:26.216385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:26.216359    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:07:26.217462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:26.217442    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:07:37.224263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.224229    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:07:37.230724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.230705    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.246615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.245168    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:07:37.246615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.245659    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e77b1d-openshift-default-dockercfg-9c6tb\""
Oct 30 19:07:37.346205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.346156    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.353113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.346536    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.361247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.361203    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.361505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.361484    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.369122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.361633    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.369390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.369364    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.369625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.369605    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.379119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.369754    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.379458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.379435    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsgc\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.990291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.968412    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.990515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.990417    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.990515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.990486    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.990679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.990660    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsgc\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.990936    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.990996    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.991050    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.991168    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.991233    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.993491    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:37.995111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:37.993731    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.015451 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.015425    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.018384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.018360    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.019138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.018974    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.021954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.021929    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.023581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.023558    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.027184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.026976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.039694 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.039670    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsgc\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.156815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.156785    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:38.536399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.536312    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:07:38.543737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.543693    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:38.543843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.543808    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:38.543890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:38.543856    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:39.247867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:39.247825    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerStarted","Data":"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"}
Oct 30 19:07:39.248072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:39.247895    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerStarted","Data":"10537d82624151f844a59222b70cd829b3eac4d02a32cda2beb566d066f28943"}
Oct 30 19:07:39.320703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:39.320635    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podStartSLOduration=2.320616588 podStartE2EDuration="2.320616588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:39.296786601 +0000 UTC m=+2079.560251557" watchObservedRunningTime="2026-10-30 19:07:39.320616588 +0000 UTC m=+2079.584081537"
Oct 30 19:07:40.173113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.169971    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:40.223450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.223415    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:40.258639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.258606    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:40.259974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.259952    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:07:40.650329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.650289    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:07:40.655345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.655326    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.670660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.670641    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:07:40.670879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.670861    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:07:40.707574 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.707543    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-bda27d1a8117\""
Oct 30 19:07:40.707975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.707942    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-hq7ks\""
Oct 30 19:07:40.708157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.708021    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-54fc64a67d3d\""
Oct 30 19:07:40.711306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.711157    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:07:40.711306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.711243    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:07:40.727635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.727521    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:07:40.838229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.838173    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.838651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.838625    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.838879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.838858    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.839131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.839112    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts92v\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.943595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.943486    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.943842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.943629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.943842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.943762    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts92v\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.943999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.943975    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.954077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.944892    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.954944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.944990    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.972713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.972576    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts92v\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.973590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.973567    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:40.975278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:40.975243    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:41.361942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:41.361917    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:07:42.308391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.308351    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" event={"ID":"ffd40fba-39de-4821-8a57-c5555e647ce9","Type":"ContainerStarted","Data":"7cf4f90351f15063560e3b05bee073257ee21521deafe9a09267503529bb95c8"}
Oct 30 19:07:42.308714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.308679    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" event={"ID":"ffd40fba-39de-4821-8a57-c5555e647ce9","Type":"ContainerStarted","Data":"708709b593cc23429587025d7d432b36143271373293632f21b84ac8119b4cb4"}
Oct 30 19:07:42.308866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.308733    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:42.310293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.310267    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:07:42.315413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.315397    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.328200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.328139    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\""
Oct 30 19:07:42.328563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.328251    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\""
Oct 30 19:07:42.328563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.328321    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\""
Oct 30 19:07:42.328563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.328378    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\""
Oct 30 19:07:42.330766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.330743    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:07:42.353575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.353527    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" podStartSLOduration=2.353514111 podStartE2EDuration="2.353514111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:42.34479493 +0000 UTC m=+2082.608259884" watchObservedRunningTime="2026-10-30 19:07:42.353514111 +0000 UTC m=+2082.616979065"
Oct 30 19:07:42.371512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.371476    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.371965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.371709    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwntz\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.371965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.371849    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.372388 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.371967    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.403111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.401447    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-af4ab7befc3d\""
Oct 30 19:07:42.407110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.405282    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-f6pzj\""
Oct 30 19:07:42.407110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.405372    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-5156225dde7c\""
Oct 30 19:07:42.473368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.473338    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwntz\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.473560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.473461    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.473638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.473568    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.473761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.473737    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.474327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.474298    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.474516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.474494    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.477478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.477457    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.510704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.510677    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwntz\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.633984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.633895    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:42.962484 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:07:42.962457    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2b3871_60ba_4f95_a879_4909c87e1743.slice/crio-4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6 WatchSource:0}: Error finding container 4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6: Status 404 returned error can't find the container with id 4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6
Oct 30 19:07:42.967783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:42.967749    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:07:43.326072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.325960    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" event={"ID":"1f2b3871-60ba-4f95-a879-4909c87e1743","Type":"ContainerStarted","Data":"2b71d2f2d75722264b7509e3bbf6e5a98a490ffe8302cdafb493bf68cc621c95"}
Oct 30 19:07:43.326072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.326030    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" event={"ID":"1f2b3871-60ba-4f95-a879-4909c87e1743","Type":"ContainerStarted","Data":"4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6"}
Oct 30 19:07:43.326367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.326348    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:43.372638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.372581    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" podStartSLOduration=1.372564133 podStartE2EDuration="1.372564133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:43.367577332 +0000 UTC m=+2083.631042309" watchObservedRunningTime="2026-10-30 19:07:43.372564133 +0000 UTC m=+2083.636029083"
Oct 30 19:07:43.464887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.464858    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:07:43.470551 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.470517    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.476653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.475625    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e77b1d\""
Oct 30 19:07:43.476653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.476432    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e77b1d-dockercfg-s47x7\""
Oct 30 19:07:43.489121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.485816    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:07:43.495344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.494956    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:07:43.512863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.512827    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.520132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.520056    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:07:43.704168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.668948    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.704168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.669215    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sk6\" (UniqueName: \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.704168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.669287    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6n4\" (UniqueName: \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.704168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.669417    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.783629    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.784842    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.784905    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.786226    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.786330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sk6\" (UniqueName: \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.786394    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6n4\" (UniqueName: \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.827022    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sk6\" (UniqueName: \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\") pod \"payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:43.925119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:43.877818    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6n4\" (UniqueName: \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\") pod \"payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:44.094040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:44.094001    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:44.135357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:44.135327    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:44.525467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:44.525175    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:07:44.530513 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:07:44.526800    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4850fc60_3a31_40ab_8325_d32f08759e7d.slice/crio-c2bd0635db6886850ab2b8b7c640f377a191f68058eea27c20a138cfd37beb31 WatchSource:0}: Error finding container c2bd0635db6886850ab2b8b7c640f377a191f68058eea27c20a138cfd37beb31: Status 404 returned error can't find the container with id c2bd0635db6886850ab2b8b7c640f377a191f68058eea27c20a138cfd37beb31
Oct 30 19:07:44.532390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:44.532361    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:07:45.359487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.359450    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" event={"ID":"71769a74-18a2-4ad1-9746-dbbcad720f92","Type":"ContainerStarted","Data":"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"}
Oct 30 19:07:45.359687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.359515    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" event={"ID":"71769a74-18a2-4ad1-9746-dbbcad720f92","Type":"ContainerStarted","Data":"8b4834d5982426efff6c82715df4b845b96808f2756d93eb92ffa3c42383c513"}
Oct 30 19:07:45.359687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.359569    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:45.361967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.361936    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" event={"ID":"4850fc60-3a31-40ab-8325-d32f08759e7d","Type":"ContainerStarted","Data":"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"}
Oct 30 19:07:45.362185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.362164    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:45.362302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.362199    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" event={"ID":"4850fc60-3a31-40ab-8325-d32f08759e7d","Type":"ContainerStarted","Data":"c2bd0635db6886850ab2b8b7c640f377a191f68058eea27c20a138cfd37beb31"}
Oct 30 19:07:45.389200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.389139    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" podStartSLOduration=2.38911937 podStartE2EDuration="2.38911937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:45.380388439 +0000 UTC m=+2085.643853416" watchObservedRunningTime="2026-10-30 19:07:45.38911937 +0000 UTC m=+2085.652584325"
Oct 30 19:07:45.405697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:45.405639    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" podStartSLOduration=2.405622585 podStartE2EDuration="2.405622585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:45.400458199 +0000 UTC m=+2085.663923149" watchObservedRunningTime="2026-10-30 19:07:45.405622585 +0000 UTC m=+2085.669087552"
Oct 30 19:07:53.104935 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:07:53.104861    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:07:53.330149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:53.330117    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:07:54.239200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.235774    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:07:54.444908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.444876    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:07:54.445076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.444927    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:07:54.445206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.445184    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.582792 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.582741    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fxd\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.602486 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602446    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.602678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602550    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.602678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602598    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.602834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602705    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.602834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602810    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.603031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602885    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.603031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602953    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.603031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.602995    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704397    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704457    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704490    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704722    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fxd\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704760    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704817    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704852    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.704912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704882    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704921    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.704925    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.705021    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.705218    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.705477    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.705581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.705504    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.707475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.707443    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.707618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.707604    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.718818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.718788    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.719349 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.719325    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fxd\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:54.908890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:54.908806    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:55.471764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:55.471735    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:07:55.477506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:55.475647    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:55.477506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:55.475717    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:55.477506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:55.475750    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:07:55.552761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:55.552736    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" event={"ID":"a7a9e791-1d37-489c-8120-1291be8e20fe","Type":"ContainerStarted","Data":"7f5d19f6d2fc821b59192d17131e9ed2cac882b4613b08f457e41c19beeba930"}
Oct 30 19:07:56.378719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.378079    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:07:56.406142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.383206    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:07:56.677165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.671408    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" event={"ID":"a7a9e791-1d37-489c-8120-1291be8e20fe","Type":"ContainerStarted","Data":"a18ea14a1f9f4a3247bd5a9bba52da3ffa8ffbb428f249683a3de23ca6bb356a"}
Oct 30 19:07:56.912112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.911417    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:07:56.912112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.911539    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:07:56.912112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:56.911586    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:07:57.921883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:57.921850    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:07:57.923348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:57.923300    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:07:58.604109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:58.603213    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:07:58.604109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:58.603296    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:07:58.929796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:58.928406    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:07:58.929796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:58.928554    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:07:59.952118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:59.946112    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:07:59.952118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:07:59.946211    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:08:00.051234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.049114    5065 generic.go:356] "Generic (PLEG): container finished" podID="3d52305f-1374-4f58-b021-693c818391bd" containerID="082c5628cc7d227e9422e9575248af0d2d88768c27467a923d3336c4d794410f" exitCode=0
Oct 30 19:08:00.260167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.256856    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerDied","Data":"082c5628cc7d227e9422e9575248af0d2d88768c27467a923d3336c4d794410f"}
Oct 30 19:08:00.260167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.257456    5065 scope.go:122] "RemoveContainer" containerID="082c5628cc7d227e9422e9575248af0d2d88768c27467a923d3336c4d794410f"
Oct 30 19:08:00.280120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.279116    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:00.280120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.279219    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:00.280120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.279294    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:00.319138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.316152    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podStartSLOduration=6.316133697 podStartE2EDuration="6.316133697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:56.716585701 +0000 UTC m=+2096.980050650" watchObservedRunningTime="2026-10-30 19:08:00.316133697 +0000 UTC m=+2100.579598648"
Oct 30 19:08:00.341474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.329336    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.488108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.487212    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:08:00.489305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.489259    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.606817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.606534    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.701110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.672853    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.701110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.674943    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.716227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.711202    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:01.000114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.955647    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:01.000114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.963582    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:08:01.000114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:00.963640    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:08:01.928111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:01.916275    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:08:01.928111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:01.916345    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:08:02.144112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:02.140774    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerStarted","Data":"b1a73aca0de029ace8053a80991b4aaca48ef1f62765204aa63ee7c65fc6a92a"}
Oct 30 19:08:02.917189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:02.909386    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 19:08:02.917189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:02.909449    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 19:08:03.005113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.004426    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:08:03.005113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.004931    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:03.005113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.005005    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:03.572008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.571973    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:08:03.590185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.590162    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:08:03.645477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.645451    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:08:03.660989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.660963    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:08:03.938942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.938857    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:08:03.939598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.939574    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:08:03.940897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:03.940878    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:08:04.013120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.012518    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:04.013120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.012573    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:04.730287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.730244    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:08:04.756826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.756792    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:08:04.757075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.756934    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.771269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.771242    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-ce5ac7999d53\""
Oct 30 19:08:04.819872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.819793    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5b29\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.820323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.820300    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.820636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.820599    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.820972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.820880    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.923690    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.923860    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5b29\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.924017    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.924133    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.925470    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.927109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.926550    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.936123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.935221    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:04.939173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:04.939141    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5b29\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:05.019108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:05.011236    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:05.019108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:05.011288    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:05.121165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:05.121123    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:05.621245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:05.621214    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:08:05.621804 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:08:05.621777    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe056a55_94e2_4658_81ef_fa67ed0ed609.slice/crio-a46d519edfa0fc7004c50ee02672e8d280b74fb9b4861455c784d60a4701447b WatchSource:0}: Error finding container a46d519edfa0fc7004c50ee02672e8d280b74fb9b4861455c784d60a4701447b: Status 404 returned error can't find the container with id a46d519edfa0fc7004c50ee02672e8d280b74fb9b4861455c784d60a4701447b
Oct 30 19:08:06.009174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:06.003424    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:06.009174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:06.003496    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:06.578290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:06.578256    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" event={"ID":"fe056a55-94e2-4658-81ef-fa67ed0ed609","Type":"ContainerStarted","Data":"845aa560f596dcbf3c7432ab88e643d31a8420da11ba50317963ed35326b7b6e"}
Oct 30 19:08:06.578517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:06.578327    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:06.578517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:06.578346    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" event={"ID":"fe056a55-94e2-4658-81ef-fa67ed0ed609","Type":"ContainerStarted","Data":"a46d519edfa0fc7004c50ee02672e8d280b74fb9b4861455c784d60a4701447b"}
Oct 30 19:08:07.007174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:07.006612    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:07.007174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:07.006676    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:08.022346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:08.008884    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body=
Oct 30 19:08:08.022346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:08.008974    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused"
Oct 30 19:08:09.010144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:09.008852    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:08:09.010326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:09.010166    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:08:09.013530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:09.011192    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:08:09.068173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:09.068124    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" podStartSLOduration=5.068107969 podStartE2EDuration="5.068107969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:06.612527972 +0000 UTC m=+2106.875992937" watchObservedRunningTime="2026-10-30 19:08:09.068107969 +0000 UTC m=+2109.331572930"
Oct 30 19:08:17.618626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:17.618514    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:08:17.694266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:17.694227    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:08:22.695600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:22.695555    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" podUID="c937ac0b-8811-4772-b54e-6bd80e522c57" containerName="payload-processing" containerID="cri-o://21ec09190a0cc6c5d2759f9475ca19f83cf601a38a92264ed18595ce05c5d1db" gracePeriod=25
Oct 30 19:08:22.849266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:22.849237    5065 generic.go:356] "Generic (PLEG): container finished" podID="c937ac0b-8811-4772-b54e-6bd80e522c57" containerID="21ec09190a0cc6c5d2759f9475ca19f83cf601a38a92264ed18595ce05c5d1db" exitCode=0
Oct 30 19:08:22.849444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:22.849312    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" event={"ID":"c937ac0b-8811-4772-b54e-6bd80e522c57","Type":"ContainerDied","Data":"21ec09190a0cc6c5d2759f9475ca19f83cf601a38a92264ed18595ce05c5d1db"}
Oct 30 19:08:23.028910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.028775    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:08:23.066809 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:23.066777    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:08:23.106047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.106018    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\") pod \"c937ac0b-8811-4772-b54e-6bd80e522c57\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") "
Oct 30 19:08:23.106403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.106066    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\") pod \"c937ac0b-8811-4772-b54e-6bd80e522c57\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") "
Oct 30 19:08:23.106504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.106410    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\") pod \"c937ac0b-8811-4772-b54e-6bd80e522c57\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") "
Oct 30 19:08:23.106585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.106565    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\") pod \"c937ac0b-8811-4772-b54e-6bd80e522c57\" (UID: \"c937ac0b-8811-4772-b54e-6bd80e522c57\") "
Oct 30 19:08:23.106692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.106639    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume" pod "c937ac0b-8811-4772-b54e-6bd80e522c57" (UID: "c937ac0b-8811-4772-b54e-6bd80e522c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:23.107515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.107492    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:23.107647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.107623    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume" pod "c937ac0b-8811-4772-b54e-6bd80e522c57" (UID: "c937ac0b-8811-4772-b54e-6bd80e522c57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:23.110886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.110832    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials" pod "c937ac0b-8811-4772-b54e-6bd80e522c57" (UID: "c937ac0b-8811-4772-b54e-6bd80e522c57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:23.110886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.110866    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw" pod "c937ac0b-8811-4772-b54e-6bd80e522c57" (UID: "c937ac0b-8811-4772-b54e-6bd80e522c57"). InnerVolumeSpecName "kube-api-access-tvdqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:23.209076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.209046    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdqw\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-kube-api-access-tvdqw\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:23.209076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.209075    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c937ac0b-8811-4772-b54e-6bd80e522c57-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:23.209342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.209130    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c937ac0b-8811-4772-b54e-6bd80e522c57-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:23.859943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.859913    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"
Oct 30 19:08:23.860456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.859955    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl" event={"ID":"c937ac0b-8811-4772-b54e-6bd80e522c57","Type":"ContainerDied","Data":"2a65deeec9939b37ad07a8153c2c67b3e1424b05e93dfcc2259c4dc3376295fb"}
Oct 30 19:08:23.860456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.860014    5065 scope.go:122] "RemoveContainer" containerID="21ec09190a0cc6c5d2759f9475ca19f83cf601a38a92264ed18595ce05c5d1db"
Oct 30 19:08:23.897472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.897448    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:08:23.906525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:23.906498    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854jbvfl"]
Oct 30 19:08:25.898867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:25.898831    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c937ac0b-8811-4772-b54e-6bd80e522c57" path="/var/lib/kubelet/pods/c937ac0b-8811-4772-b54e-6bd80e522c57/volumes"
Oct 30 19:08:33.488530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:33.487890    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:08:33.586107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:33.583338    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:33.674283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:33.673187    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:37.258400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.258354    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:37.260071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.260049    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c937ac0b-8811-4772-b54e-6bd80e522c57" containerName="payload-processing"
Oct 30 19:08:37.260071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.260069    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c937ac0b-8811-4772-b54e-6bd80e522c57" containerName="payload-processing"
Oct 30 19:08:37.260664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.260649    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c937ac0b-8811-4772-b54e-6bd80e522c57" containerName="payload-processing"
Oct 30 19:08:37.309155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.305685    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:37.309155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.305825    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.319327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.319296    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:08:37.319482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.319295    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:08:37.319482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.319332    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-4d813f9bc72b\""
Oct 30 19:08:37.319482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.319296    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:08:37.319482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.319366    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:08:37.378194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.378158    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.378369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.378348    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rq9v\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.378529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.378513    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.378650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.378637    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.486505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.486460    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.486872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.486850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rq9v\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.487166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.487143    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.491322 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.491291    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.511162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.493799    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.540110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.514158    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.564117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.563358    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.579983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.579941    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rq9v\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:37.666750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:37.666719    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:38.045837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.044912    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:08:38.070159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.059671    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.077369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.071289    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-94b1b3df02aa\""
Oct 30 19:08:38.116341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.116173    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:08:38.136889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.136855    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.137058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.136977    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.137058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.137024    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.137555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.137247    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncs4\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.238999    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.239163    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.239212    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.246212    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.247003    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.249112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.247403    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncs4\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.428614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.410207    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.497484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.497407    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncs4\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.532281 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:08:38.522495    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce752fd6_5b27_4a46_85de_7b6dd02e5bbc.slice/crio-e830d03f3f4dd62a1adb2136f61308175cde3cbb72b3bb972ed7ddac9fb8b915 WatchSource:0}: Error finding container e830d03f3f4dd62a1adb2136f61308175cde3cbb72b3bb972ed7ddac9fb8b915: Status 404 returned error can't find the container with id e830d03f3f4dd62a1adb2136f61308175cde3cbb72b3bb972ed7ddac9fb8b915
Oct 30 19:08:38.564434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.564407    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:38.749163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.749127    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:38.769139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.769076    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" event={"ID":"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc","Type":"ContainerStarted","Data":"e830d03f3f4dd62a1adb2136f61308175cde3cbb72b3bb972ed7ddac9fb8b915"}
Oct 30 19:08:38.960074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:38.956442    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:39.017331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.008371    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.023861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.023834    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:39.025561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.025526    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"routing-overlay\""
Oct 30 19:08:39.027434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.027415    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"agc-cred-5c1227258e85\""
Oct 30 19:08:39.028794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.028545    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-e2e--307b9dfd-dockercfg-2n6p7\""
Oct 30 19:08:39.028794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.028603    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8c554\""
Oct 30 19:08:39.028794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.028674    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"kube-root-ca.crt\""
Oct 30 19:08:39.028794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.028764    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"openshift-service-ca.crt\""
Oct 30 19:08:39.203229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.172394    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.203575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.203548    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.237116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.226334    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpxd\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.237656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.237621    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.360705 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.360663    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.360885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.360743    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.360885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.360862    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpxd\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.375392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.360991    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.375392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.361810    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.379006 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.378976    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.527118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.519836    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpxd\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.554386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.531643    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:39.677540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.662899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:08:39.741297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:39.678137    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:40.023695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:40.023660    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" event={"ID":"bff4de91-b684-4342-a2f1-52541d43ba76","Type":"ContainerStarted","Data":"58248cae13e5803d0bc91ce39bf599f7829f1fafdc1c877491cf3019cd79214b"}
Oct 30 19:08:40.023975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:40.023955    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" event={"ID":"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc","Type":"ContainerStarted","Data":"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"}
Oct 30 19:08:40.071679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:40.071614    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" podStartSLOduration=3.071594995 podStartE2EDuration="3.071594995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:40.040742486 +0000 UTC m=+2140.304207434" watchObservedRunningTime="2026-10-30 19:08:40.071594995 +0000 UTC m=+2140.335059944"
Oct 30 19:08:41.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:41.330610    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" event={"ID":"bff4de91-b684-4342-a2f1-52541d43ba76","Type":"ContainerStarted","Data":"27b8bbf6b1eb7395f7d2ec2ba719eb2ac8a1465ee2955b1aea8bd29b54eca3b5"}
Oct 30 19:08:41.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:41.330686    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:41.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:41.330742    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:41.347069 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:08:41.341917    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a713f24_07cb_4ffe_a824_9e3768280e58.slice/crio-19f6d7054d47e76cb8851c5ce0a058ae9dd782b3ef52981f79744a2cbd3a7a74 WatchSource:0}: Error finding container 19f6d7054d47e76cb8851c5ce0a058ae9dd782b3ef52981f79744a2cbd3a7a74: Status 404 returned error can't find the container with id 19f6d7054d47e76cb8851c5ce0a058ae9dd782b3ef52981f79744a2cbd3a7a74
Oct 30 19:08:41.347069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:41.346225    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:41.355370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:41.355326    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" podStartSLOduration=4.355313528 podStartE2EDuration="4.355313528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:41.354710245 +0000 UTC m=+2141.618175194" watchObservedRunningTime="2026-10-30 19:08:41.355313528 +0000 UTC m=+2141.618778476"
Oct 30 19:08:42.333134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:42.332870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" event={"ID":"3a713f24-07cb-4ffe-a824-9e3768280e58","Type":"ContainerStarted","Data":"6a953a49c3aaefbd7ebf06252811fb0a8790bdb1734c07f30a4e699d1fabadf9"}
Oct 30 19:08:42.333134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:42.332949    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" event={"ID":"3a713f24-07cb-4ffe-a824-9e3768280e58","Type":"ContainerStarted","Data":"19f6d7054d47e76cb8851c5ce0a058ae9dd782b3ef52981f79744a2cbd3a7a74"}
Oct 30 19:08:42.334687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:42.333607    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:42.366400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:42.366350    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" podStartSLOduration=4.366337388 podStartE2EDuration="4.366337388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:42.363726804 +0000 UTC m=+2142.627191769" watchObservedRunningTime="2026-10-30 19:08:42.366337388 +0000 UTC m=+2142.629802335"
Oct 30 19:08:44.212899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.212871    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:44.306070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.306023    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"]
Oct 30 19:08:44.338644 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.338482    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"]
Oct 30 19:08:44.338644 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.338618    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.344193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.343974    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:08:44.344739 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.344681    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:08:44.365901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.358241    5065 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" secret="" err="secret \"payload-processing-external-model-e2e--307b9dfd-dockercfg-2n6p7\" not found"
Oct 30 19:08:44.431194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.399524    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhsvg\" (UniqueName: \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.431194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.399609    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.431194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.399686    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.688666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.663677    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhsvg\" (UniqueName: \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.688666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.687231    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.688666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.687335    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.698567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.688756    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.698567 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:44.686125    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-5c1227258e85: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:44.698567 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:44.689044    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:44.698567 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:44.689141    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:45.189122252 +0000 UTC m=+2145.452587200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:44.716391 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:44.702361    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:44.716391 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:44.702421    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:45.20240333 +0000 UTC m=+2145.465868267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:44.806991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.806918    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhsvg\" (UniqueName: \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:44.819770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:44.811920    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1078e303-xhcmd\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:45.108281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:45.081476    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:45.239896 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:45.239453    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-5c1227258e85: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:45.239896 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:45.239496    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:45.239896 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:45.239585    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:46.239563849 +0000 UTC m=+2146.503028798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:45.240625 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:45.240558    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:45.240705 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:45.240636    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:46.24062128 +0000 UTC m=+2146.504086227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:45.768543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:45.766597    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:45.773510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:45.772370    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:46.245626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:46.243845    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"]
Oct 30 19:08:46.270879 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:46.269243    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:46.271206 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:46.271185    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:48.271163444 +0000 UTC m=+2148.534628390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:46.271822 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:46.271799    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-5c1227258e85: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:46.271991 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:46.271974    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:46.351372 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:46.272171    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:48.272154067 +0000 UTC m=+2148.535619009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:46.490153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:46.490026    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd" event={"ID":"c378a181-b022-4b5b-bebc-3f755f257cfe","Type":"ContainerStarted","Data":"d4b5c084fb4df95c0c1278e5b1d72645d9254fec2ed9b49df875226d7ddadef1"}
Oct 30 19:08:47.049653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:47.049617    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"]
Oct 30 19:08:47.064987 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:47.064955    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6942ce87-qk2jv"]
Oct 30 19:08:47.507884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:47.507790    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd" event={"ID":"c378a181-b022-4b5b-bebc-3f755f257cfe","Type":"ContainerStarted","Data":"7a12c472e1ec696963ee44cb73ff35c831179fd5b6366a10811a4f2d02499110"}
Oct 30 19:08:47.540725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:47.540672    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd" podStartSLOduration=3.540657393 podStartE2EDuration="3.540657393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:47.535904366 +0000 UTC m=+2147.799369314" watchObservedRunningTime="2026-10-30 19:08:47.540657393 +0000 UTC m=+2147.804122340"
Oct 30 19:08:48.171850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:48.171817    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fdb365-7e71-4f28-8265-106b10237ac7" path="/var/lib/kubelet/pods/82fdb365-7e71-4f28-8265-106b10237ac7/volumes"
Oct 30 19:08:48.343886 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:48.343847    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-5c1227258e85: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:48.343886 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:48.343878    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq: secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:48.344196 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:48.343947    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:52.343927438 +0000 UTC m=+2152.607392381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : secret "agc-cred-5c1227258e85" not found
Oct 30 19:08:48.344196 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:48.344022    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:48.344196 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:48.344137    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume podName:3a713f24-07cb-4ffe-a824-9e3768280e58 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:52.344077852 +0000 UTC m=+2152.607542795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:08:49.591440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:49.591406    5065 generic.go:356] "Generic (PLEG): container finished" podID="c378a181-b022-4b5b-bebc-3f755f257cfe" containerID="7a12c472e1ec696963ee44cb73ff35c831179fd5b6366a10811a4f2d02499110" exitCode=0
Oct 30 19:08:49.656587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:49.591496    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd" event={"ID":"c378a181-b022-4b5b-bebc-3f755f257cfe","Type":"ContainerDied","Data":"7a12c472e1ec696963ee44cb73ff35c831179fd5b6366a10811a4f2d02499110"}
Oct 30 19:08:50.408824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.408752    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" podUID="3a713f24-07cb-4ffe-a824-9e3768280e58" containerName="payload-processing" containerID="cri-o://6a953a49c3aaefbd7ebf06252811fb0a8790bdb1734c07f30a4e699d1fabadf9" gracePeriod=25
Oct 30 19:08:50.611227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.610369    5065 generic.go:356] "Generic (PLEG): container finished" podID="3a713f24-07cb-4ffe-a824-9e3768280e58" containerID="6a953a49c3aaefbd7ebf06252811fb0a8790bdb1734c07f30a4e699d1fabadf9" exitCode=0
Oct 30 19:08:50.611227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.610943    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" event={"ID":"3a713f24-07cb-4ffe-a824-9e3768280e58","Type":"ContainerDied","Data":"6a953a49c3aaefbd7ebf06252811fb0a8790bdb1734c07f30a4e699d1fabadf9"}
Oct 30 19:08:50.767865 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.767818    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" podUID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" containerName="payload-processing" containerID="cri-o://2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a" gracePeriod=25
Oct 30 19:08:50.920017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.919954    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:50.960353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:50.960296    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:51.109634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.109598    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\") pod \"3a713f24-07cb-4ffe-a824-9e3768280e58\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") "
Oct 30 19:08:51.110002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.109978    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\" (UniqueName: \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\") pod \"c378a181-b022-4b5b-bebc-3f755f257cfe\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") "
Oct 30 19:08:51.110267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.110249    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\") pod \"3a713f24-07cb-4ffe-a824-9e3768280e58\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") "
Oct 30 19:08:51.110502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.110486    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\") pod \"c378a181-b022-4b5b-bebc-3f755f257cfe\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") "
Oct 30 19:08:51.110627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.110612    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\") pod \"c378a181-b022-4b5b-bebc-3f755f257cfe\" (UID: \"c378a181-b022-4b5b-bebc-3f755f257cfe\") "
Oct 30 19:08:51.132408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.110840    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\") pod \"3a713f24-07cb-4ffe-a824-9e3768280e58\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") "
Oct 30 19:08:51.132408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.110906    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\") pod \"3a713f24-07cb-4ffe-a824-9e3768280e58\" (UID: \"3a713f24-07cb-4ffe-a824-9e3768280e58\") "
Oct 30 19:08:51.153804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.153463    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd" pod "3a713f24-07cb-4ffe-a824-9e3768280e58" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58"). InnerVolumeSpecName "kube-api-access-sfpxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:51.162191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.161733    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume" pod "3a713f24-07cb-4ffe-a824-9e3768280e58" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:51.162191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.161747    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca" pod "c378a181-b022-4b5b-bebc-3f755f257cfe" (UID: "c378a181-b022-4b5b-bebc-3f755f257cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:51.162191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.162163    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume" pod "3a713f24-07cb-4ffe-a824-9e3768280e58" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:51.174336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.172595    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg" pod "c378a181-b022-4b5b-bebc-3f755f257cfe" (UID: "c378a181-b022-4b5b-bebc-3f755f257cfe"). InnerVolumeSpecName "kube-api-access-vhsvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:51.174336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.173931    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca" pod "c378a181-b022-4b5b-bebc-3f755f257cfe" (UID: "c378a181-b022-4b5b-bebc-3f755f257cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:51.175750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.175371    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials" pod "3a713f24-07cb-4ffe-a824-9e3768280e58" (UID: "3a713f24-07cb-4ffe-a824-9e3768280e58"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:51.177382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.177357    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:08:51.178846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.178821    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3a713f24-07cb-4ffe-a824-9e3768280e58" containerName="payload-processing"
Oct 30 19:08:51.178968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.178848    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a713f24-07cb-4ffe-a824-9e3768280e58" containerName="payload-processing"
Oct 30 19:08:51.178968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.178962    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c378a181-b022-4b5b-bebc-3f755f257cfe" containerName="revoke-keys"
Oct 30 19:08:51.179200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.178970    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c378a181-b022-4b5b-bebc-3f755f257cfe" containerName="revoke-keys"
Oct 30 19:08:51.179318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.179217    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c378a181-b022-4b5b-bebc-3f755f257cfe" containerName="revoke-keys"
Oct 30 19:08:51.179318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.179235    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a713f24-07cb-4ffe-a824-9e3768280e58" containerName="payload-processing"
Oct 30 19:08:51.204745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.204253    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:51.214617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.214592    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:08:51.215061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.215041    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.217498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.217408    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\") pod \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") "
Oct 30 19:08:51.218141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.217668    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\") pod \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") "
Oct 30 19:08:51.218141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.217752    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\") pod \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") "
Oct 30 19:08:51.218350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218157    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218221    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218290    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218388    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218425    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218490    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218583    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218743    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2s5\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218794    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218892    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhsvg\" (UniqueName: \"kubernetes.io/projected/c378a181-b022-4b5b-bebc-3f755f257cfe-kube-api-access-vhsvg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218908    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218923    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218935    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c378a181-b022-4b5b-bebc-3f755f257cfe-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218948    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a713f24-07cb-4ffe-a824-9e3768280e58-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218959    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.218972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.218970    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpxd\" (UniqueName: \"kubernetes.io/projected/3a713f24-07cb-4ffe-a824-9e3768280e58-kube-api-access-sfpxd\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.219899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.219863    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume" pod "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" (UID: "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:51.222077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.222060    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v" pod "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" (UID: "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc"). InnerVolumeSpecName "kube-api-access-5rq9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:51.222335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.222316    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials" pod "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" (UID: "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:51.231572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.231231    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-21d896-openshift-default-dockercfg-xdx6m\""
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.319710    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\") pod \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\" (UID: \"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc\") "
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.319971    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320020    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320052    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320135    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320191    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320212    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320261    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320317    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320384    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2s5\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320448    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rq9v\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-kube-api-access-5rq9v\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320459    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.320870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.320469    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.321738    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.321970    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.322271    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.322543    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.323403    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.323599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume" pod "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" (UID: "ce752fd6-5b27-4a46-85de-7b6dd02e5bbc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.324520    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.329831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.324779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.347815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.347794    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.347979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.347828    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2s5\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.422202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.422125    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:51.628264 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.551358    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:51.637366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.636921    5065 generic.go:356] "Generic (PLEG): container finished" podID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" containerID="2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a" exitCode=0
Oct 30 19:08:51.637366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.637052    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" event={"ID":"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc","Type":"ContainerDied","Data":"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"}
Oct 30 19:08:51.637366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.637176    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l" event={"ID":"ce752fd6-5b27-4a46-85de-7b6dd02e5bbc","Type":"ContainerDied","Data":"e830d03f3f4dd62a1adb2136f61308175cde3cbb72b3bb972ed7ddac9fb8b915"}
Oct 30 19:08:51.637366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.637217    5065 scope.go:122] "RemoveContainer" containerID="2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.637634    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.639768    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq" event={"ID":"3a713f24-07cb-4ffe-a824-9e3768280e58","Type":"ContainerDied","Data":"19f6d7054d47e76cb8851c5ce0a058ae9dd782b3ef52981f79744a2cbd3a7a74"}
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.639912    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.642364    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd" event={"ID":"c378a181-b022-4b5b-bebc-3f755f257cfe","Type":"ContainerDied","Data":"d4b5c084fb4df95c0c1278e5b1d72645d9254fec2ed9b49df875226d7ddadef1"}
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.642434    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b5c084fb4df95c0c1278e5b1d72645d9254fec2ed9b49df875226d7ddadef1"
Oct 30 19:08:51.675448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.642523    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"
Oct 30 19:08:51.710482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.710199    5065 scope.go:122] "RemoveContainer" containerID="2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"
Oct 30 19:08:51.711371 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:51.711339    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a\": container with ID starting with 2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a not found: ID does not exist" containerID="2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"
Oct 30 19:08:51.711496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.711393    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a"} err="failed to get container status \"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a\": rpc error: code = NotFound desc = could not find container \"2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a\": container with ID starting with 2c184df669437a2809fbf75ea9f335105db6cde51f9ef78ef16320cddd01b51a not found: ID does not exist"
Oct 30 19:08:51.711496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.711425    5065 scope.go:122] "RemoveContainer" containerID="6a953a49c3aaefbd7ebf06252811fb0a8790bdb1734c07f30a4e699d1fabadf9"
Oct 30 19:08:51.716659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.716571    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:51.725720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.725021    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-78f5bdtfq"]
Oct 30 19:08:51.739528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.739505    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:51.746434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.746413    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-77ffskp2l"]
Oct 30 19:08:51.896458 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:08:51.895852    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8926e2_79ba_45d0_b6d3_34e8a013b052.slice/crio-e0b60c4cff84d96357b2d4ac74b4e33bcebdfb14801b5343a7773c1248836cf9 WatchSource:0}: Error finding container e0b60c4cff84d96357b2d4ac74b4e33bcebdfb14801b5343a7773c1248836cf9: Status 404 returned error can't find the container with id e0b60c4cff84d96357b2d4ac74b4e33bcebdfb14801b5343a7773c1248836cf9
Oct 30 19:08:51.902995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.902957    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:51.903129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.903048    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:51.903185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.903135    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:51.908868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.907906    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a713f24-07cb-4ffe-a824-9e3768280e58" path="/var/lib/kubelet/pods/3a713f24-07cb-4ffe-a824-9e3768280e58/volumes"
Oct 30 19:08:51.908868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.908591    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" path="/var/lib/kubelet/pods/ce752fd6-5b27-4a46-85de-7b6dd02e5bbc/volumes"
Oct 30 19:08:51.909518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:51.909108    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:08:52.336539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.336513    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:08:52.500536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.499876    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:08:52.654195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.654114    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerDied","Data":"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"}
Oct 30 19:08:52.654607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.654077    5065 generic.go:356] "Generic (PLEG): container finished" podID="e10d8444-b860-4111-8190-718915129e0e" containerID="8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a" exitCode=0
Oct 30 19:08:52.654607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.654587    5065 scope.go:122] "RemoveContainer" containerID="8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"
Oct 30 19:08:52.656828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.656793    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:52.657480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.656886    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:52.657480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.656938    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:08:52.659495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.659470    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" event={"ID":"8e8926e2-79ba-45d0-b6d3-34e8a013b052","Type":"ContainerStarted","Data":"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"}
Oct 30 19:08:52.659608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.659519    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" event={"ID":"8e8926e2-79ba-45d0-b6d3-34e8a013b052","Type":"ContainerStarted","Data":"e0b60c4cff84d96357b2d4ac74b4e33bcebdfb14801b5343a7773c1248836cf9"}
Oct 30 19:08:52.731762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.731705    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podStartSLOduration=1.731680333 podStartE2EDuration="1.731680333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:52.724120335 +0000 UTC m=+2152.987585283" watchObservedRunningTime="2026-10-30 19:08:52.731680333 +0000 UTC m=+2152.995145284"
Oct 30 19:08:52.943890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:52.940824    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:08:53.133721 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:08:53.133691    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:08:53.552353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:53.552318    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:08:53.553257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:53.552733    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:53.553257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:53.552765    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:53.670751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:53.670714    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerStarted","Data":"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"}
Oct 30 19:08:54.157619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:54.157586    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:08:54.159094 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:54.159060    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:54.159154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:54.159129    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:54.560598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:54.560548    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:54.560828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:54.560615    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:55.157566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:55.157532    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:55.158106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:55.157590    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:55.259310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:55.259282    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:08:55.553121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:55.552973    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:55.553121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:55.553044    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:56.172589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:56.160461    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:56.172589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:56.160533    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:56.410110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:56.405994    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:08:56.554567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:56.554527    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:56.554875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:56.554844    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:57.188109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.179295    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:57.188109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.179478    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:57.502301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.502168    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" podUID="ffd40fba-39de-4821-8a57-c5555e647ce9" containerName="payload-processing" containerID="cri-o://7cf4f90351f15063560e3b05bee073257ee21521deafe9a09267503529bb95c8" gracePeriod=25
Oct 30 19:08:57.592363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.568214    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:57.592363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.568297    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:57.945054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:57.943167    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" podUID="1f2b3871-60ba-4f95-a879-4909c87e1743" containerName="payload-processing" containerID="cri-o://2b71d2f2d75722264b7509e3bbf6e5a98a490ffe8302cdafb493bf68cc621c95" gracePeriod=25
Oct 30 19:08:58.224192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.224076    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:58.311221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.311175    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:58.362699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.362663    5065 generic.go:356] "Generic (PLEG): container finished" podID="ffd40fba-39de-4821-8a57-c5555e647ce9" containerID="7cf4f90351f15063560e3b05bee073257ee21521deafe9a09267503529bb95c8" exitCode=0
Oct 30 19:08:58.363317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.363292    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" event={"ID":"ffd40fba-39de-4821-8a57-c5555e647ce9","Type":"ContainerDied","Data":"7cf4f90351f15063560e3b05bee073257ee21521deafe9a09267503529bb95c8"}
Oct 30 19:08:58.388805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.388769    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"]
Oct 30 19:08:58.414822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.414767    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07eff279-pmrdw"]
Oct 30 19:08:58.538265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.538222    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.546285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.540348    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.584568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.584510    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.594941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.594907    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:58.595231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.595203    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:58.641332 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.641291    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.669161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.668225    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.742121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.711311    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.882170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:58.877653    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:08:59.181162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.176502    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:08:59.181162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.176589    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:08:59.485668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.484416    5065 generic.go:356] "Generic (PLEG): container finished" podID="1f2b3871-60ba-4f95-a879-4909c87e1743" containerID="2b71d2f2d75722264b7509e3bbf6e5a98a490ffe8302cdafb493bf68cc621c95" exitCode=0
Oct 30 19:08:59.485668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.484769    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" event={"ID":"1f2b3871-60ba-4f95-a879-4909c87e1743","Type":"ContainerDied","Data":"2b71d2f2d75722264b7509e3bbf6e5a98a490ffe8302cdafb493bf68cc621c95"}
Oct 30 19:08:59.563115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.549986    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.563115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.550579    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.593660    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.594071    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.632457    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.632804    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.634314    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:08:59.641111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.634367    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:08:59.737170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.723363    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.737170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.723982    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:08:59.737170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.724334    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:08:59.737170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.724395    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:08:59.804193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.803163    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.804193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.803564    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:08:59.815345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.807515    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:08:59.952749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:08:59.913356    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:09:00.568289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.564186    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:09:00.568289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.564312    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:09:00.762145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.759051    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 19:09:00.764189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.762442    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 19:09:00.827167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.781052    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef7aa3a-3dc0-4b61-9076-9e680758c292" path="/var/lib/kubelet/pods/5ef7aa3a-3dc0-4b61-9076-9e680758c292/volumes"
Oct 30 19:09:00.846637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.846589    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk" event={"ID":"1f2b3871-60ba-4f95-a879-4909c87e1743","Type":"ContainerDied","Data":"4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6"}
Oct 30 19:09:00.846826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.846669    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4048103d048e4c36f7ebf0233f47709634894718d93825f31b224dfcaa1ba3f6"
Oct 30 19:09:00.849826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.849803    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:09:00.850235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850210    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:09:00.850333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850273    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:09:00.850333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850289    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm" event={"ID":"ffd40fba-39de-4821-8a57-c5555e647ce9","Type":"ContainerDied","Data":"708709b593cc23429587025d7d432b36143271373293632f21b84ac8119b4cb4"}
Oct 30 19:09:00.850333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850321    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708709b593cc23429587025d7d432b36143271373293632f21b84ac8119b4cb4"
Oct 30 19:09:00.850587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850352    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:09:00.850587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.850408    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:09:00.860795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.860773    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:09:00.860978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.860966    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:09:00.861496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.861477    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:09:00.871251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.871229    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:09:00.958662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.958627    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:09:00.964562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.964535    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\") pod \"1f2b3871-60ba-4f95-a879-4909c87e1743\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") "
Oct 30 19:09:00.964844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.964801    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\") pod \"ffd40fba-39de-4821-8a57-c5555e647ce9\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") "
Oct 30 19:09:00.964971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.964862    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\") pod \"1f2b3871-60ba-4f95-a879-4909c87e1743\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") "
Oct 30 19:09:00.964971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.964910    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\") pod \"ffd40fba-39de-4821-8a57-c5555e647ce9\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") "
Oct 30 19:09:00.965142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.965010    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\") pod \"ffd40fba-39de-4821-8a57-c5555e647ce9\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") "
Oct 30 19:09:00.965142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.965057    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\") pod \"ffd40fba-39de-4821-8a57-c5555e647ce9\" (UID: \"ffd40fba-39de-4821-8a57-c5555e647ce9\") "
Oct 30 19:09:00.965142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.965126    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\") pod \"1f2b3871-60ba-4f95-a879-4909c87e1743\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") "
Oct 30 19:09:00.965354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.965282    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\") pod \"1f2b3871-60ba-4f95-a879-4909c87e1743\" (UID: \"1f2b3871-60ba-4f95-a879-4909c87e1743\") "
Oct 30 19:09:00.966804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.966778    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume" pod "ffd40fba-39de-4821-8a57-c5555e647ce9" (UID: "ffd40fba-39de-4821-8a57-c5555e647ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:00.967891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.967863    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume" pod "ffd40fba-39de-4821-8a57-c5555e647ce9" (UID: "ffd40fba-39de-4821-8a57-c5555e647ce9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:00.969760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.969738    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume" pod "1f2b3871-60ba-4f95-a879-4909c87e1743" (UID: "1f2b3871-60ba-4f95-a879-4909c87e1743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:00.970906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.969943    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume" pod "1f2b3871-60ba-4f95-a879-4909c87e1743" (UID: "1f2b3871-60ba-4f95-a879-4909c87e1743"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:00.970906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:00.970158    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials" pod "ffd40fba-39de-4821-8a57-c5555e647ce9" (UID: "ffd40fba-39de-4821-8a57-c5555e647ce9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:01.063564 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.063522    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz" pod "1f2b3871-60ba-4f95-a879-4909c87e1743" (UID: "1f2b3871-60ba-4f95-a879-4909c87e1743"). InnerVolumeSpecName "kube-api-access-fwntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:01.064219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.064189    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v" pod "ffd40fba-39de-4821-8a57-c5555e647ce9" (UID: "ffd40fba-39de-4821-8a57-c5555e647ce9"). InnerVolumeSpecName "kube-api-access-ts92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.099933    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials" pod "1f2b3871-60ba-4f95-a879-4909c87e1743" (UID: "1f2b3871-60ba-4f95-a879-4909c87e1743"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100639    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100664    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwntz\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-kube-api-access-fwntz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100682    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100695    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f2b3871-60ba-4f95-a879-4909c87e1743-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100710    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd40fba-39de-4821-8a57-c5555e647ce9-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100723    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100736    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts92v\" (UniqueName: \"kubernetes.io/projected/ffd40fba-39de-4821-8a57-c5555e647ce9-kube-api-access-ts92v\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.113174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.100752    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2b3871-60ba-4f95-a879-4909c87e1743-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:01.160108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.159477    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:09:01.160108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.159535    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:09:01.454070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:01.453958    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" podUID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" containerName="payload-processing" containerID="cri-o://29cd0cb60da2cb2765b2b5a19084cbcb1486bad3d79c8ed99bb7e502e69e70a8" gracePeriod=25
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.097613    5065 generic.go:356] "Generic (PLEG): container finished" podID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" containerID="29cd0cb60da2cb2765b2b5a19084cbcb1486bad3d79c8ed99bb7e502e69e70a8" exitCode=0
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.098078    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.100345    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" event={"ID":"4adfcb30-a158-4a1c-8646-a0844b1ff6c0","Type":"ContainerDied","Data":"29cd0cb60da2cb2765b2b5a19084cbcb1486bad3d79c8ed99bb7e502e69e70a8"}
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.100473    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.104703    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"
Oct 30 19:09:02.106170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.105369    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:09:02.128360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.128166    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:09:02.161100 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.158336    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 19:09:02.161100 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.158406    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 19:09:02.214141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.203636    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:09:02.214141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.208317    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5hdsqk"]
Oct 30 19:09:02.322325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.310170    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:09:02.322325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.312259    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8bsqjm"]
Oct 30 19:09:02.684206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.660651    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.828761    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\") pod \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") "
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.829233    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\") pod \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") "
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.829359    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\") pod \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") "
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.829410    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\") pod \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\" (UID: \"4adfcb30-a158-4a1c-8646-a0844b1ff6c0\") "
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.830646    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume" pod "4adfcb30-a158-4a1c-8646-a0844b1ff6c0" (UID: "4adfcb30-a158-4a1c-8646-a0844b1ff6c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:02.836344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.831836    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume" pod "4adfcb30-a158-4a1c-8646-a0844b1ff6c0" (UID: "4adfcb30-a158-4a1c-8646-a0844b1ff6c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:02.840798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.840769    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials" pod "4adfcb30-a158-4a1c-8646-a0844b1ff6c0" (UID: "4adfcb30-a158-4a1c-8646-a0844b1ff6c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:02.842295 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.842265    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84" pod "4adfcb30-a158-4a1c-8646-a0844b1ff6c0" (UID: "4adfcb30-a158-4a1c-8646-a0844b1ff6c0"). InnerVolumeSpecName "kube-api-access-9zv84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:02.933001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.932965    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zv84\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-kube-api-access-9zv84\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:02.933001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.933000    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:02.933297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.933014    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:02.933297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:02.933029    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4adfcb30-a158-4a1c-8646-a0844b1ff6c0-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:03.110726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.110692    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"
Oct 30 19:09:03.111203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.110719    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m" event={"ID":"4adfcb30-a158-4a1c-8646-a0844b1ff6c0","Type":"ContainerDied","Data":"a6e8e954c86f1fb2dd0d82ed387126eab1a0a905ee77f274dbfae5e8f312407a"}
Oct 30 19:09:03.111203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.110789    5065 scope.go:122] "RemoveContainer" containerID="29cd0cb60da2cb2765b2b5a19084cbcb1486bad3d79c8ed99bb7e502e69e70a8"
Oct 30 19:09:03.173920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.173885    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:09:03.194841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.194809    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-9cbdlfh7m"]
Oct 30 19:09:03.263857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.263827    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:09:03.265550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.265527    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:09:03.893439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.893409    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2b3871-60ba-4f95-a879-4909c87e1743" path="/var/lib/kubelet/pods/1f2b3871-60ba-4f95-a879-4909c87e1743/volumes"
Oct 30 19:09:03.893886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.893870    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" path="/var/lib/kubelet/pods/4adfcb30-a158-4a1c-8646-a0844b1ff6c0/volumes"
Oct 30 19:09:03.894296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:03.894280    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd40fba-39de-4821-8a57-c5555e647ce9" path="/var/lib/kubelet/pods/ffd40fba-39de-4821-8a57-c5555e647ce9/volumes"
Oct 30 19:09:05.959332 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:05.959299    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" podUID="fe056a55-94e2-4658-81ef-fa67ed0ed609" containerName="payload-processing" containerID="cri-o://845aa560f596dcbf3c7432ab88e643d31a8420da11ba50317963ed35326b7b6e" gracePeriod=25
Oct 30 19:09:06.168203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.168139    5065 generic.go:356] "Generic (PLEG): container finished" podID="fe056a55-94e2-4658-81ef-fa67ed0ed609" containerID="845aa560f596dcbf3c7432ab88e643d31a8420da11ba50317963ed35326b7b6e" exitCode=0
Oct 30 19:09:06.168467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.168447    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" event={"ID":"fe056a55-94e2-4658-81ef-fa67ed0ed609","Type":"ContainerDied","Data":"845aa560f596dcbf3c7432ab88e643d31a8420da11ba50317963ed35326b7b6e"}
Oct 30 19:09:06.275002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.274974    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:09:06.336248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.336217    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\") pod \"fe056a55-94e2-4658-81ef-fa67ed0ed609\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") "
Oct 30 19:09:06.336450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.336321    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\") pod \"fe056a55-94e2-4658-81ef-fa67ed0ed609\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") "
Oct 30 19:09:06.336450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.336349    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\") pod \"fe056a55-94e2-4658-81ef-fa67ed0ed609\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") "
Oct 30 19:09:06.336450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.336379    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\") pod \"fe056a55-94e2-4658-81ef-fa67ed0ed609\" (UID: \"fe056a55-94e2-4658-81ef-fa67ed0ed609\") "
Oct 30 19:09:06.337178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.337144    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume" pod "fe056a55-94e2-4658-81ef-fa67ed0ed609" (UID: "fe056a55-94e2-4658-81ef-fa67ed0ed609"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:06.337350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.337327    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume" pod "fe056a55-94e2-4658-81ef-fa67ed0ed609" (UID: "fe056a55-94e2-4658-81ef-fa67ed0ed609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:06.339508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.339477    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29" pod "fe056a55-94e2-4658-81ef-fa67ed0ed609" (UID: "fe056a55-94e2-4658-81ef-fa67ed0ed609"). InnerVolumeSpecName "kube-api-access-x5b29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:06.339650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.339526    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials" pod "fe056a55-94e2-4658-81ef-fa67ed0ed609" (UID: "fe056a55-94e2-4658-81ef-fa67ed0ed609"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:06.438763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.438723    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:06.438763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.438760    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe056a55-94e2-4658-81ef-fa67ed0ed609-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:06.438763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.438770    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:06.438763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:06.438779    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5b29\" (UniqueName: \"kubernetes.io/projected/fe056a55-94e2-4658-81ef-fa67ed0ed609-kube-api-access-x5b29\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:07.181183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.181146    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd" event={"ID":"fe056a55-94e2-4658-81ef-fa67ed0ed609","Type":"ContainerDied","Data":"a46d519edfa0fc7004c50ee02672e8d280b74fb9b4861455c784d60a4701447b"}
Oct 30 19:09:07.181650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.181227    5065 scope.go:122] "RemoveContainer" containerID="845aa560f596dcbf3c7432ab88e643d31a8420da11ba50317963ed35326b7b6e"
Oct 30 19:09:07.181650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.181157    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"
Oct 30 19:09:07.214760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.214705    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:09:07.219822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.219800    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdqbknd"]
Oct 30 19:09:07.895959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:07.895926    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe056a55-94e2-4658-81ef-fa67ed0ed609" path="/var/lib/kubelet/pods/fe056a55-94e2-4658-81ef-fa67ed0ed609/volumes"
Oct 30 19:09:12.788731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.788697    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:09:12.791422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.791394    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ffd40fba-39de-4821-8a57-c5555e647ce9" containerName="payload-processing"
Oct 30 19:09:12.791589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.791574    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd40fba-39de-4821-8a57-c5555e647ce9" containerName="payload-processing"
Oct 30 19:09:12.791765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.791752    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" containerName="payload-processing"
Oct 30 19:09:12.791851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.791840    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" containerName="payload-processing"
Oct 30 19:09:12.792046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.792032    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1f2b3871-60ba-4f95-a879-4909c87e1743" containerName="payload-processing"
Oct 30 19:09:12.792189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.792176    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2b3871-60ba-4f95-a879-4909c87e1743" containerName="payload-processing"
Oct 30 19:09:12.796316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.796293    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fe056a55-94e2-4658-81ef-fa67ed0ed609" containerName="payload-processing"
Oct 30 19:09:12.796450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.796437    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe056a55-94e2-4658-81ef-fa67ed0ed609" containerName="payload-processing"
Oct 30 19:09:12.796586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.796576    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" containerName="payload-processing"
Oct 30 19:09:12.796665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.796653    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" containerName="payload-processing"
Oct 30 19:09:12.797217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.797197    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd40fba-39de-4821-8a57-c5555e647ce9" containerName="payload-processing"
Oct 30 19:09:12.797440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.797424    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce752fd6-5b27-4a46-85de-7b6dd02e5bbc" containerName="payload-processing"
Oct 30 19:09:12.797652 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.797638    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2b3871-60ba-4f95-a879-4909c87e1743" containerName="payload-processing"
Oct 30 19:09:12.797767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.797754    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe056a55-94e2-4658-81ef-fa67ed0ed609" containerName="payload-processing"
Oct 30 19:09:12.798255 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.797882    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adfcb30-a158-4a1c-8646-a0844b1ff6c0" containerName="payload-processing"
Oct 30 19:09:12.817374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.808263    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:12.817374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.816398    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"e2e-cmp-legacy-00d613-provider-cred\""
Oct 30 19:09:12.817374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.816479    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-plugins-e2e-extmod-a-de4bda\""
Oct 30 19:09:12.817374 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.816517    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:09:12.820957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.820881    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"openshift-service-ca.crt\""
Oct 30 19:09:12.821106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.820968    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"kube-root-ca.crt\""
Oct 30 19:09:12.821266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.821242    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"routing-overlay\""
Oct 30 19:09:12.946672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.946625    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:12.947458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.947427    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq2v\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:12.947727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.947708    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:12.955283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:12.955253    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.063168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.062618    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.069118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.063527    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.069118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.064613    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.069118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.065160    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.069118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.066368    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.069118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.066566    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq2v\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.088334 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.083858    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.136114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.120800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq2v\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.147108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.141320    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:13.719035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:13.718548    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:09:14.530129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:14.528045    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" event={"ID":"d108de79-a3c8-4bb8-a687-692d9e8e7bc1","Type":"ContainerStarted","Data":"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"}
Oct 30 19:09:14.530129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:14.528138    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" event={"ID":"d108de79-a3c8-4bb8-a687-692d9e8e7bc1","Type":"ContainerStarted","Data":"d6a0ff89b94a19a199a2d05e54655fdd7be1439e96c80fd5532867a040ccda73"}
Oct 30 19:09:14.530129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:14.529477    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:14.560496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:14.560440    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" podStartSLOduration=2.560426382 podStartE2EDuration="2.560426382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:14.554910328 +0000 UTC m=+2174.818375293" watchObservedRunningTime="2026-10-30 19:09:14.560426382 +0000 UTC m=+2174.823891330"
Oct 30 19:09:22.101110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:22.095165    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"]
Oct 30 19:09:22.140115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:22.134128    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"]
Oct 30 19:09:22.155188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:22.155152    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f11969-xzdzp"]
Oct 30 19:09:22.202213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:22.202136    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10cfdb0-tzk46"]
Oct 30 19:09:22.288317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:22.287704    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:09:23.039072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:23.039041    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"]
Oct 30 19:09:23.045176 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:23.045152    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be2d3ede-pmmvb"]
Oct 30 19:09:23.075133 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:09:23.072329    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:09:23.891996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:23.891965    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f30ffbd-d331-4948-ac30-24c1392a6041" path="/var/lib/kubelet/pods/0f30ffbd-d331-4948-ac30-24c1392a6041/volumes"
Oct 30 19:09:23.892543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:23.892516    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f58e5e-0179-4950-9bbc-0d9dc258790c" path="/var/lib/kubelet/pods/d0f58e5e-0179-4950-9bbc-0d9dc258790c/volumes"
Oct 30 19:09:23.893029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:23.893014    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc3895a-03e2-4ff7-b77a-016f38d72ca1" path="/var/lib/kubelet/pods/ffc3895a-03e2-4ff7-b77a-016f38d72ca1/volumes"
Oct 30 19:09:26.551372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.551342    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:09:26.717638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.717560    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:09:26.725826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.725799    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.734824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.734803    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-616d58511ea0\""
Oct 30 19:09:26.739617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.739589    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:09:26.762096 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.762050    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.762226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.762177    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhmb\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.762341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.762324    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.762475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.762453    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.864614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.864536    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.864764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.864636    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhmb\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.864816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.864770    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.864913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.864885    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.865453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.865427    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.865561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.865526    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.867447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.867427    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:26.877693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:26.877672    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhmb\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:27.047934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.047900    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:27.614128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.609730    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:09:27.722386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.719930    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.722386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.721446    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:09:27.728725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.726355    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:09:27.730343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.730259    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-f6pzj\""
Oct 30 19:09:27.738799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738527    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\""
Oct 30 19:09:27.738799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738611    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-af4ab7befc3d\""
Oct 30 19:09:27.738799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738655    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-5156225dde7c\""
Oct 30 19:09:27.738799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738701    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\""
Oct 30 19:09:27.738799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738718    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\""
Oct 30 19:09:27.739121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.738902    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\""
Oct 30 19:09:27.808380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.808337    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.808578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.808438    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.808645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.808584    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.808712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.808653    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh5r\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910165    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910267    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910445    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910575    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh5r\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910950    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.912023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.910984    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.913697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.913676    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.920476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.920443    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" event={"ID":"fb073d66-ef5f-43ec-a96b-f1c664810e7e","Type":"ContainerStarted","Data":"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"}
Oct 30 19:09:27.921218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.920509    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" event={"ID":"fb073d66-ef5f-43ec-a96b-f1c664810e7e","Type":"ContainerStarted","Data":"de81dc524da6d12f9909770867cec20714b838b53994a0b0b65008b71ce3df57"}
Oct 30 19:09:27.921218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.920560    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:27.924858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.924836    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh5r\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:27.946656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:27.946618    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" podStartSLOduration=1.946608436 podStartE2EDuration="1.946608436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:27.944232404 +0000 UTC m=+2188.207697382" watchObservedRunningTime="2026-10-30 19:09:27.946608436 +0000 UTC m=+2188.210073384"
Oct 30 19:09:28.067452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.067421    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:28.452231 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:09:28.452200    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918ca963_86b1_4f95_b2a2_232602cbf59c.slice/crio-7507393149e4ed81db285f9fcc2dca75588d67b98e94143f9db32a27cfcb910e WatchSource:0}: Error finding container 7507393149e4ed81db285f9fcc2dca75588d67b98e94143f9db32a27cfcb910e: Status 404 returned error can't find the container with id 7507393149e4ed81db285f9fcc2dca75588d67b98e94143f9db32a27cfcb910e
Oct 30 19:09:28.453605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.453581    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:09:28.933035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.932995    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" event={"ID":"918ca963-86b1-4f95-b2a2-232602cbf59c","Type":"ContainerStarted","Data":"740d418b5ce93fe9169114581ff14ea9bce35dbeeb9606f447e604ca8d12600f"}
Oct 30 19:09:28.933565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.933058    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" event={"ID":"918ca963-86b1-4f95-b2a2-232602cbf59c","Type":"ContainerStarted","Data":"7507393149e4ed81db285f9fcc2dca75588d67b98e94143f9db32a27cfcb910e"}
Oct 30 19:09:28.933565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.933330    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:28.958613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:28.958562    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" podStartSLOduration=1.9585373000000001 podStartE2EDuration="1.9585373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:28.954150961 +0000 UTC m=+2189.217615930" watchObservedRunningTime="2026-10-30 19:09:28.9585373 +0000 UTC m=+2189.222002253"
Oct 30 19:09:32.031816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:32.031782    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"]
Oct 30 19:09:32.037570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:32.037543    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c8827c-4sr64"]
Oct 30 19:09:32.041971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:32.041951    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"]
Oct 30 19:09:32.046249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:32.046223    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d666c3da-4kw8m"]
Oct 30 19:09:33.076119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:33.072619    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"]
Oct 30 19:09:33.100563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:33.100521    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b693a587-kqt2k"]
Oct 30 19:09:33.919857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:33.919814    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c09306-7b14-4e1b-a8a1-9ae9bad9969c" path="/var/lib/kubelet/pods/51c09306-7b14-4e1b-a8a1-9ae9bad9969c/volumes"
Oct 30 19:09:33.921275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:33.920618    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83945599-919f-4a4c-8a7b-d65a09a6da5f" path="/var/lib/kubelet/pods/83945599-919f-4a4c-8a7b-d65a09a6da5f/volumes"
Oct 30 19:09:33.921275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:33.921216    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a" path="/var/lib/kubelet/pods/fb8c9ef1-b4e1-4d9a-b5f4-d157ea95677a/volumes"
Oct 30 19:09:36.578438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.578408    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:09:36.585487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.585462    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.595103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.595055    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4e94aa-openshift-default-dockercfg-ffhn5\""
Oct 30 19:09:36.597939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.597911    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:09:36.733333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733299    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.733510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733437    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.733569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733558    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.733604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733583    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.734386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733615    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.734386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733680    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.734386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733742    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.734386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733807    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwd4\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.734386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.733868    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835036    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835110    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835162    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835211    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835265    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835340    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwd4\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835395    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835488    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835757    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835901    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835906    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.835969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.835969    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.836405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.836376    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.838059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.838037    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.838595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.838574    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.852764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.852738    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwd4\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.856245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.856222    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:36.908977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:36.908958    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:37.218857 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:09:37.218813    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf139ad_7550_4b83_a8ff_4c8cc95a96b5.slice/crio-e3ace449c60f9dc0c57cfd1c9636fa5d9c4d88a95957c2b9b3248f00097da838 WatchSource:0}: Error finding container e3ace449c60f9dc0c57cfd1c9636fa5d9c4d88a95957c2b9b3248f00097da838: Status 404 returned error can't find the container with id e3ace449c60f9dc0c57cfd1c9636fa5d9c4d88a95957c2b9b3248f00097da838
Oct 30 19:09:37.222080 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:37.222046    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:09:37.222513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:37.222479    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:09:37.222635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:37.222562    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:09:37.222635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:37.222609    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:09:38.038687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.038654    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" event={"ID":"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5","Type":"ContainerStarted","Data":"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"}
Oct 30 19:09:38.039239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.038716    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" event={"ID":"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5","Type":"ContainerStarted","Data":"e3ace449c60f9dc0c57cfd1c9636fa5d9c4d88a95957c2b9b3248f00097da838"}
Oct 30 19:09:38.073701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.073579    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podStartSLOduration=2.07356065 podStartE2EDuration="2.07356065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:38.06858688 +0000 UTC m=+2198.332051876" watchObservedRunningTime="2026-10-30 19:09:38.07356065 +0000 UTC m=+2198.337025599"
Oct 30 19:09:38.956057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.956028    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:38.973753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.956406    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:38.974072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:38.974038    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:39.014112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:39.013806    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:09:39.168999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:39.168964    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:09:39.917113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:39.916492    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:39.917113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:39.916547    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:39.945381 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:39.945357    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:09:40.913405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:40.913312    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:40.913992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:40.913958    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:41.915112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:41.914292    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:41.915112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:41.914362    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:42.910414 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:42.910380    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:42.910715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:42.910691    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:43.914154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:43.914121    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 19:09:43.914670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:43.914334    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 19:09:44.180166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:44.172205    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" podUID="bff4de91-b684-4342-a2f1-52541d43ba76" containerName="payload-processing" containerID="cri-o://27b8bbf6b1eb7395f7d2ec2ba719eb2ac8a1465ee2955b1aea8bd29b54eca3b5" gracePeriod=25
Oct 30 19:09:44.374113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:44.373611    5065 generic.go:356] "Generic (PLEG): container finished" podID="bff4de91-b684-4342-a2f1-52541d43ba76" containerID="27b8bbf6b1eb7395f7d2ec2ba719eb2ac8a1465ee2955b1aea8bd29b54eca3b5" exitCode=0
Oct 30 19:09:44.374113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:44.373666    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" event={"ID":"bff4de91-b684-4342-a2f1-52541d43ba76","Type":"ContainerDied","Data":"27b8bbf6b1eb7395f7d2ec2ba719eb2ac8a1465ee2955b1aea8bd29b54eca3b5"}
Oct 30 19:09:45.275285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.269702    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:09:45.324111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.312833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\") pod \"bff4de91-b684-4342-a2f1-52541d43ba76\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") "
Oct 30 19:09:45.324471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.324437    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\") pod \"bff4de91-b684-4342-a2f1-52541d43ba76\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") "
Oct 30 19:09:45.324822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.324798    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\") pod \"bff4de91-b684-4342-a2f1-52541d43ba76\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") "
Oct 30 19:09:45.329119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.325167    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\") pod \"bff4de91-b684-4342-a2f1-52541d43ba76\" (UID: \"bff4de91-b684-4342-a2f1-52541d43ba76\") "
Oct 30 19:09:45.367298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.352415    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4" pod "bff4de91-b684-4342-a2f1-52541d43ba76" (UID: "bff4de91-b684-4342-a2f1-52541d43ba76"). InnerVolumeSpecName "kube-api-access-sncs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:45.372889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.352764    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume" pod "bff4de91-b684-4342-a2f1-52541d43ba76" (UID: "bff4de91-b684-4342-a2f1-52541d43ba76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:45.373068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.371945    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume" pod "bff4de91-b684-4342-a2f1-52541d43ba76" (UID: "bff4de91-b684-4342-a2f1-52541d43ba76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:45.383025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.382957    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials" pod "bff4de91-b684-4342-a2f1-52541d43ba76" (UID: "bff4de91-b684-4342-a2f1-52541d43ba76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:45.475145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.474231    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:45.475145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.474266    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sncs4\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-kube-api-access-sncs4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:45.475145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.474282    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bff4de91-b684-4342-a2f1-52541d43ba76-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:45.475145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.474299    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bff4de91-b684-4342-a2f1-52541d43ba76-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:45.508790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.508746    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5" event={"ID":"bff4de91-b684-4342-a2f1-52541d43ba76","Type":"ContainerDied","Data":"58248cae13e5803d0bc91ce39bf599f7829f1fafdc1c877491cf3019cd79214b"}
Oct 30 19:09:45.509116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.509075    5065 scope.go:122] "RemoveContainer" containerID="27b8bbf6b1eb7395f7d2ec2ba719eb2ac8a1465ee2955b1aea8bd29b54eca3b5"
Oct 30 19:09:45.509402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:45.509386    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"
Oct 30 19:09:46.261331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.261270    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:46.261716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.261644    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:46.417305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.417274    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:09:46.428226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.422517    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b588bj5"]
Oct 30 19:09:46.991108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.987473    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:46.991108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.988297    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:46.995337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:46.994917    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:09:47.896532 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:09:47.896498    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff4de91-b684-4342-a2f1-52541d43ba76" path="/var/lib/kubelet/pods/bff4de91-b684-4342-a2f1-52541d43ba76/volumes"
Oct 30 19:09:53.207344 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:09:53.207299    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:10:00.982330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.982287    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"]
Oct 30 19:10:00.983519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.983494    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bff4de91-b684-4342-a2f1-52541d43ba76" containerName="payload-processing"
Oct 30 19:10:00.983519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.983518    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff4de91-b684-4342-a2f1-52541d43ba76" containerName="payload-processing"
Oct 30 19:10:00.983787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.983771    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff4de91-b684-4342-a2f1-52541d43ba76" containerName="payload-processing"
Oct 30 19:10:00.988573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.988543    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:00.994796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.994737    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:00.997100 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.997053    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:00.997504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:00.997481    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"]
Oct 30 19:10:01.220923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.220885    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.221159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.221008    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlzx\" (UniqueName: \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.221427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.221409    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.324189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.324116    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.324405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.324252    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.324405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.324313    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlzx\" (UniqueName: \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.325152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.325009    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.326995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.326966    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.339450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.339410    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlzx\" (UniqueName: \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\") pod \"maas-api-revoke-sub-7d4af195-ccdgf\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.636573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.636472    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:01.935616 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:01.935569    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae69d73_d4de_4bf3_b58d_20327c90ce0d.slice/crio-5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97 WatchSource:0}: Error finding container 5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97: Status 404 returned error can't find the container with id 5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97
Oct 30 19:10:01.936389 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:01.936326    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"]
Oct 30 19:10:02.854614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:02.854559    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf" event={"ID":"cae69d73-d4de-4bf3-b58d-20327c90ce0d","Type":"ContainerStarted","Data":"e11001d4d6df8925ec7099af3edfb04e48834d77f3725c94905ce85690726fe0"}
Oct 30 19:10:02.855263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:02.854643    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf" event={"ID":"cae69d73-d4de-4bf3-b58d-20327c90ce0d","Type":"ContainerStarted","Data":"5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97"}
Oct 30 19:10:02.877765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:02.877700    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf" podStartSLOduration=2.877681048 podStartE2EDuration="2.877681048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:02.876751689 +0000 UTC m=+2223.140216649" watchObservedRunningTime="2026-10-30 19:10:02.877681048 +0000 UTC m=+2223.141146008"
Oct 30 19:10:03.899621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:03.899402    5065 generic.go:356] "Generic (PLEG): container finished" podID="cae69d73-d4de-4bf3-b58d-20327c90ce0d" containerID="e11001d4d6df8925ec7099af3edfb04e48834d77f3725c94905ce85690726fe0" exitCode=0
Oct 30 19:10:03.901664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:03.901626    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf" event={"ID":"cae69d73-d4de-4bf3-b58d-20327c90ce0d","Type":"ContainerDied","Data":"e11001d4d6df8925ec7099af3edfb04e48834d77f3725c94905ce85690726fe0"}
Oct 30 19:10:05.165495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.165412    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:05.346350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.346318    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\") pod \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") "
Oct 30 19:10:05.346558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.346444    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\" (UniqueName: \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\") pod \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") "
Oct 30 19:10:05.347261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.346567    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\") pod \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\" (UID: \"cae69d73-d4de-4bf3-b58d-20327c90ce0d\") "
Oct 30 19:10:05.347261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.346708    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca" pod "cae69d73-d4de-4bf3-b58d-20327c90ce0d" (UID: "cae69d73-d4de-4bf3-b58d-20327c90ce0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:05.347466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.347444    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:05.348820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.348792    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca" pod "cae69d73-d4de-4bf3-b58d-20327c90ce0d" (UID: "cae69d73-d4de-4bf3-b58d-20327c90ce0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:05.348914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.348882    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx" pod "cae69d73-d4de-4bf3-b58d-20327c90ce0d" (UID: "cae69d73-d4de-4bf3-b58d-20327c90ce0d"). InnerVolumeSpecName "kube-api-access-jmlzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:05.449691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.449608    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmlzx\" (UniqueName: \"kubernetes.io/projected/cae69d73-d4de-4bf3-b58d-20327c90ce0d-kube-api-access-jmlzx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:05.449691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.449641    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae69d73-d4de-4bf3-b58d-20327c90ce0d-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:05.681051 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.681011    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:10:05.923480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.923440    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf" event={"ID":"cae69d73-d4de-4bf3-b58d-20327c90ce0d","Type":"ContainerDied","Data":"5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97"}
Oct 30 19:10:05.923671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.923498    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ecdd6520e1c45f916361beed283139635c8315bc6b18e6f814b1886f051bb97"
Oct 30 19:10:05.923671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:05.923559    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"
Oct 30 19:10:10.683430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:10.683193    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" podUID="918ca963-86b1-4f95-b2a2-232602cbf59c" containerName="payload-processing" containerID="cri-o://740d418b5ce93fe9169114581ff14ea9bce35dbeeb9606f447e604ca8d12600f" gracePeriod=25
Oct 30 19:10:10.978760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:10.978670    5065 generic.go:356] "Generic (PLEG): container finished" podID="918ca963-86b1-4f95-b2a2-232602cbf59c" containerID="740d418b5ce93fe9169114581ff14ea9bce35dbeeb9606f447e604ca8d12600f" exitCode=0
Oct 30 19:10:10.979071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:10.979002    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" event={"ID":"918ca963-86b1-4f95-b2a2-232602cbf59c","Type":"ContainerDied","Data":"740d418b5ce93fe9169114581ff14ea9bce35dbeeb9606f447e604ca8d12600f"}
Oct 30 19:10:11.056278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.056254    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:10:11.222706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.221701    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\") pod \"918ca963-86b1-4f95-b2a2-232602cbf59c\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") "
Oct 30 19:10:11.222706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.221795    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\") pod \"918ca963-86b1-4f95-b2a2-232602cbf59c\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") "
Oct 30 19:10:11.222706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.222018    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\") pod \"918ca963-86b1-4f95-b2a2-232602cbf59c\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") "
Oct 30 19:10:11.222706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.222219    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\") pod \"918ca963-86b1-4f95-b2a2-232602cbf59c\" (UID: \"918ca963-86b1-4f95-b2a2-232602cbf59c\") "
Oct 30 19:10:11.223824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.223591    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume" pod "918ca963-86b1-4f95-b2a2-232602cbf59c" (UID: "918ca963-86b1-4f95-b2a2-232602cbf59c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:11.223824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.223623    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume" pod "918ca963-86b1-4f95-b2a2-232602cbf59c" (UID: "918ca963-86b1-4f95-b2a2-232602cbf59c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:11.225827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.225800    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials" pod "918ca963-86b1-4f95-b2a2-232602cbf59c" (UID: "918ca963-86b1-4f95-b2a2-232602cbf59c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:11.226358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.226334    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r" pod "918ca963-86b1-4f95-b2a2-232602cbf59c" (UID: "918ca963-86b1-4f95-b2a2-232602cbf59c"). InnerVolumeSpecName "kube-api-access-vxh5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:11.325091 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.325043    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:11.325225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.325097    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxh5r\" (UniqueName: \"kubernetes.io/projected/918ca963-86b1-4f95-b2a2-232602cbf59c-kube-api-access-vxh5r\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:11.325225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.325110    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:11.325225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.325118    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/918ca963-86b1-4f95-b2a2-232602cbf59c-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:11.852716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.852680    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"]
Oct 30 19:10:11.853554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853538    5065 cpu_manager.go:414] "containerMap: removing container" podUID="918ca963-86b1-4f95-b2a2-232602cbf59c" containerName="payload-processing"
Oct 30 19:10:11.853621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853555    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ca963-86b1-4f95-b2a2-232602cbf59c" containerName="payload-processing"
Oct 30 19:10:11.853621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853591    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cae69d73-d4de-4bf3-b58d-20327c90ce0d" containerName="revoke-keys"
Oct 30 19:10:11.853621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853596    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae69d73-d4de-4bf3-b58d-20327c90ce0d" containerName="revoke-keys"
Oct 30 19:10:11.853726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853720    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ca963-86b1-4f95-b2a2-232602cbf59c" containerName="payload-processing"
Oct 30 19:10:11.853764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.853736    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae69d73-d4de-4bf3-b58d-20327c90ce0d" containerName="revoke-keys"
Oct 30 19:10:11.857896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.857875    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:11.862612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.862590    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:11.865140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.864852    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:11.866174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.866148    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"]
Oct 30 19:10:11.993455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.993429    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"
Oct 30 19:10:11.993455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.993441    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm" event={"ID":"918ca963-86b1-4f95-b2a2-232602cbf59c","Type":"ContainerDied","Data":"7507393149e4ed81db285f9fcc2dca75588d67b98e94143f9db32a27cfcb910e"}
Oct 30 19:10:11.993682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:11.993508    5065 scope.go:122] "RemoveContainer" containerID="740d418b5ce93fe9169114581ff14ea9bce35dbeeb9606f447e604ca8d12600f"
Oct 30 19:10:12.020330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.020300    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:10:12.025440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.025417    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-99b5r5tsm"]
Oct 30 19:10:12.037397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.037368    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.037609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.037586    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.037665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.037629    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25q8\" (UniqueName: \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.139475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.139409    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.139588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.139512    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.139588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.139544    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25q8\" (UniqueName: \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.140213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.140193    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.142001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.141983    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.153360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.153340    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25q8\" (UniqueName: \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\") pod \"maas-api-revoke-sub-f5e25053-pf8x4\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.176666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.176644    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:12.539576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.537840    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"]
Oct 30 19:10:12.541193 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:12.539974    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a4bb9f_94f8_4052_8bc8_4d5298fe97e9.slice/crio-02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01 WatchSource:0}: Error finding container 02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01: Status 404 returned error can't find the container with id 02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01
Oct 30 19:10:12.634019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.633991    5065 scope.go:122] "RemoveContainer" containerID="9455acaf755b88ee3ad6d80cbe7aecdf33ac0442dcb91e882568a07199340a19"
Oct 30 19:10:12.646589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:12.646569    5065 scope.go:122] "RemoveContainer" containerID="96fd5fa55b7d24a178d840204a00d9d3417361e3e8b5a170344aa994053007a6"
Oct 30 19:10:13.007146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:13.007073    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4" event={"ID":"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9","Type":"ContainerStarted","Data":"02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01"}
Oct 30 19:10:13.897353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:13.897317    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918ca963-86b1-4f95-b2a2-232602cbf59c" path="/var/lib/kubelet/pods/918ca963-86b1-4f95-b2a2-232602cbf59c/volumes"
Oct 30 19:10:14.021169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:14.021134    5065 generic.go:356] "Generic (PLEG): container finished" podID="57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" containerID="b966427aad779f7069914ca24b1ffb172de66e99cf6349dc8c525ef60d69e73b" exitCode=0
Oct 30 19:10:14.021567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:14.021212    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4" event={"ID":"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9","Type":"ContainerDied","Data":"b966427aad779f7069914ca24b1ffb172de66e99cf6349dc8c525ef60d69e73b"}
Oct 30 19:10:15.305886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.305826    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:15.419033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.419003    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\" (UniqueName: \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\") pod \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") "
Oct 30 19:10:15.419407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.419386    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\") pod \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") "
Oct 30 19:10:15.419502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.419427    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\") pod \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\" (UID: \"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9\") "
Oct 30 19:10:15.419836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.419804    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca" pod "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" (UID: "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:15.420266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.420232    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.422329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.422301    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca" pod "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" (UID: "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:15.422435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.422411    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8" pod "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" (UID: "57a4bb9f-94f8-4052-8bc8-4d5298fe97e9"). InnerVolumeSpecName "kube-api-access-m25q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:15.522196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.522165    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.522196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:15.522194    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25q8\" (UniqueName: \"kubernetes.io/projected/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9-kube-api-access-m25q8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:16.074531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:16.074494    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4" event={"ID":"57a4bb9f-94f8-4052-8bc8-4d5298fe97e9","Type":"ContainerDied","Data":"02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01"}
Oct 30 19:10:16.074743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:16.074551    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02729abb2bf275df02abdad63fd31d0d919c14b2f143af0c1507b97ad761df01"
Oct 30 19:10:16.074743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:16.074617    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"
Oct 30 19:10:17.496110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:17.486963    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 19:10:17.496110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:17.487236    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" containerID="cri-o://666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd" gracePeriod=30
Oct 30 19:10:22.926482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:22.926405    5065 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:10:22.926482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:22.926456    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:10:23.061065 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:10:23.061033    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:10:23.314116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.309915    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 19:10:23.342446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.342403    5065 generic.go:356] "Generic (PLEG): container finished" podID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerID="666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd" exitCode=0
Oct 30 19:10:23.342729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.342675    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" event={"ID":"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34","Type":"ContainerDied","Data":"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"}
Oct 30 19:10:23.342902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.342735    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f" event={"ID":"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34","Type":"ContainerDied","Data":"45e5ccd776508b9c9fac2db499e6d2e49fb23aa63bd5b8f850516f63b4e47c24"}
Oct 30 19:10:23.342902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.342767    5065 scope.go:122] "RemoveContainer" containerID="666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"
Oct 30 19:10:23.343029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.342941    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"
Oct 30 19:10:23.362269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.362230    5065 scope.go:122] "RemoveContainer" containerID="666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"
Oct 30 19:10:23.362982 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:10:23.362945    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd\": container with ID starting with 666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd not found: ID does not exist" containerID="666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"
Oct 30 19:10:23.363070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.362996    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd"} err="failed to get container status \"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd\": rpc error: code = NotFound desc = could not find container \"666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd\": container with ID starting with 666f79d968becc5d407c523647c9f17fef0602d93191eb3ba45997e838a1aebd not found: ID does not exist"
Oct 30 19:10:23.429934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.429901    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430075    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430208    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430231    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:23.430319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430273    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430303    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430381    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430468    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:23.430579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430526    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430599    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.430855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.430644    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\") pod \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\" (UID: \"1bbcd5f2-48e7-4a14-a2b9-56a3e847af34\") "
Oct 30 19:10:23.431351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.431312    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:23.431841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.431694    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.431841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.431720    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.431841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.431734    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.432172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.432141    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:23.432325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.432306    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:23.433626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.433478    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:23.433819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.433795    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "kube-api-access-lqm96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:23.434713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.434686    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:23.436561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.436527    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token" pod "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" (UID: "1bbcd5f2-48e7-4a14-a2b9-56a3e847af34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:23.533517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533490    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.533517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533515    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.533744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533527    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.533744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533535    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.533744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533545    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.533744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.533553    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqm96\" (UniqueName: \"kubernetes.io/projected/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34-kube-api-access-lqm96\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:23.713886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.713846    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 19:10:23.734408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.734382    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-gqz4f"]
Oct 30 19:10:23.895579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:23.895493    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" path="/var/lib/kubelet/pods/1bbcd5f2-48e7-4a14-a2b9-56a3e847af34/volumes"
Oct 30 19:10:24.458565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.455840    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"]
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458581    5065 cpu_manager.go:414] "containerMap: removing container" podUID="57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" containerName="revoke-keys"
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458604    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" containerName="revoke-keys"
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458715    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy"
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458725    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy"
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458924    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbcd5f2-48e7-4a14-a2b9-56a3e847af34" containerName="istio-proxy"
Oct 30 19:10:24.459041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.458938    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" containerName="revoke-keys"
Oct 30 19:10:24.470116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.464191    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.471905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.471880    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:24.472219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.472201    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:24.475513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.475486    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"]
Oct 30 19:10:24.578745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.578715    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.579027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.579009    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4hs\" (UniqueName: \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.579163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.579043    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.681170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.681138    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4hs\" (UniqueName: \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.681326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.681180    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.681444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.681325    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.682031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.681999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.683927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.683893    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.695581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.695553    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4hs\" (UniqueName: \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\") pod \"maas-api-revoke-sub-c1c5deb4-kzfwx\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:24.783778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:24.783757    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:25.085755 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:25.085726    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd92e381_0bab_4637_a96d_7f1c9bf13935.slice/crio-e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2 WatchSource:0}: Error finding container e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2: Status 404 returned error can't find the container with id e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2
Oct 30 19:10:25.086956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:25.086932    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"]
Oct 30 19:10:25.399025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:25.398953    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx" event={"ID":"cd92e381-0bab-4637-a96d-7f1c9bf13935","Type":"ContainerStarted","Data":"e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2"}
Oct 30 19:10:26.413034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:26.413002    5065 generic.go:356] "Generic (PLEG): container finished" podID="cd92e381-0bab-4637-a96d-7f1c9bf13935" containerID="c2a2f857cc3a20153d205c91ed4cefbde5e7240793683408a1a43664daf924d8" exitCode=0
Oct 30 19:10:26.413549 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:26.413159    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx" event={"ID":"cd92e381-0bab-4637-a96d-7f1c9bf13935","Type":"ContainerDied","Data":"c2a2f857cc3a20153d205c91ed4cefbde5e7240793683408a1a43664daf924d8"}
Oct 30 19:10:27.624926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.624901    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:27.765199 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.765014    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\" (UniqueName: \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\") pod \"cd92e381-0bab-4637-a96d-7f1c9bf13935\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") "
Oct 30 19:10:27.765376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.765258    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\") pod \"cd92e381-0bab-4637-a96d-7f1c9bf13935\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") "
Oct 30 19:10:27.765462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.765370    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\") pod \"cd92e381-0bab-4637-a96d-7f1c9bf13935\" (UID: \"cd92e381-0bab-4637-a96d-7f1c9bf13935\") "
Oct 30 19:10:27.766654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.766620    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca" pod "cd92e381-0bab-4637-a96d-7f1c9bf13935" (UID: "cd92e381-0bab-4637-a96d-7f1c9bf13935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:27.769619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.769585    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca" pod "cd92e381-0bab-4637-a96d-7f1c9bf13935" (UID: "cd92e381-0bab-4637-a96d-7f1c9bf13935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:27.770221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.770197    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:27.770333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.770227    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd92e381-0bab-4637-a96d-7f1c9bf13935-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:27.781796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.781766    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs" pod "cd92e381-0bab-4637-a96d-7f1c9bf13935" (UID: "cd92e381-0bab-4637-a96d-7f1c9bf13935"). InnerVolumeSpecName "kube-api-access-7h4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:27.872610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:27.872567    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h4hs\" (UniqueName: \"kubernetes.io/projected/cd92e381-0bab-4637-a96d-7f1c9bf13935-kube-api-access-7h4hs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:28.435393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.435366    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"
Oct 30 19:10:28.435599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.435365    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx" event={"ID":"cd92e381-0bab-4637-a96d-7f1c9bf13935","Type":"ContainerDied","Data":"e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2"}
Oct 30 19:10:28.435599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.435492    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9df59b4e973e9257aad1cb078cbedcb1435cfe33b2285c047455031825c75b2"
Oct 30 19:10:28.832984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.832953    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"]
Oct 30 19:10:28.833924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.833905    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cd92e381-0bab-4637-a96d-7f1c9bf13935" containerName="revoke-keys"
Oct 30 19:10:28.833924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.833924    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd92e381-0bab-4637-a96d-7f1c9bf13935" containerName="revoke-keys"
Oct 30 19:10:28.834127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.834075    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd92e381-0bab-4637-a96d-7f1c9bf13935" containerName="revoke-keys"
Oct 30 19:10:28.840783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.838441    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:28.845582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.845450    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:28.845582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.845556    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:28.850667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.850638    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"]
Oct 30 19:10:28.932162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.932122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqpf\" (UniqueName: \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:28.932357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.932251    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:28.932675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:28.932651    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.034327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.034295    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.034546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.034394    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqpf\" (UniqueName: \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.034546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.034457    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.034948 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.034925    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.037233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.037209    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.048386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.048362    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqpf\" (UniqueName: \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\") pod \"maas-api-revoke-sub-71212b3c-mnzfj\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.175375 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.175303    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:29.475596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:29.475567    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"]
Oct 30 19:10:29.477130 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:29.477101    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d684e4d_b989_4b39_b532_d5ac310147d8.slice/crio-e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a WatchSource:0}: Error finding container e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a: Status 404 returned error can't find the container with id e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a
Oct 30 19:10:30.464483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:30.464442    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj" event={"ID":"5d684e4d-b989-4b39-b532-d5ac310147d8","Type":"ContainerStarted","Data":"272bff93d830b3fee5365eea5707b69151dff15c2143d7ce1661d66a0aaf2261"}
Oct 30 19:10:30.464981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:30.464510    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj" event={"ID":"5d684e4d-b989-4b39-b532-d5ac310147d8","Type":"ContainerStarted","Data":"e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a"}
Oct 30 19:10:30.495463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:30.495415    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj" podStartSLOduration=2.495401179 podStartE2EDuration="2.495401179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:30.484961088 +0000 UTC m=+2250.748426037" watchObservedRunningTime="2026-10-30 19:10:30.495401179 +0000 UTC m=+2250.758866126"
Oct 30 19:10:31.482941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:31.482899    5065 generic.go:356] "Generic (PLEG): container finished" podID="5d684e4d-b989-4b39-b532-d5ac310147d8" containerID="272bff93d830b3fee5365eea5707b69151dff15c2143d7ce1661d66a0aaf2261" exitCode=0
Oct 30 19:10:31.483455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:31.483060    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj" event={"ID":"5d684e4d-b989-4b39-b532-d5ac310147d8","Type":"ContainerDied","Data":"272bff93d830b3fee5365eea5707b69151dff15c2143d7ce1661d66a0aaf2261"}
Oct 30 19:10:32.680862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.680835    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:32.816837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.816808    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\" (UniqueName: \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\") pod \"5d684e4d-b989-4b39-b532-d5ac310147d8\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") "
Oct 30 19:10:32.817044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.816945    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\") pod \"5d684e4d-b989-4b39-b532-d5ac310147d8\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") "
Oct 30 19:10:32.817528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.817140    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\") pod \"5d684e4d-b989-4b39-b532-d5ac310147d8\" (UID: \"5d684e4d-b989-4b39-b532-d5ac310147d8\") "
Oct 30 19:10:32.817827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.817580    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca" pod "5d684e4d-b989-4b39-b532-d5ac310147d8" (UID: "5d684e4d-b989-4b39-b532-d5ac310147d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:32.818528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.818493    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:32.819804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.819574    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf" pod "5d684e4d-b989-4b39-b532-d5ac310147d8" (UID: "5d684e4d-b989-4b39-b532-d5ac310147d8"). InnerVolumeSpecName "kube-api-access-4bqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:32.821356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.821321    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca" pod "5d684e4d-b989-4b39-b532-d5ac310147d8" (UID: "5d684e4d-b989-4b39-b532-d5ac310147d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:32.945115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.941472    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqpf\" (UniqueName: \"kubernetes.io/projected/5d684e4d-b989-4b39-b532-d5ac310147d8-kube-api-access-4bqpf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:32.945115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:32.941503    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d684e4d-b989-4b39-b532-d5ac310147d8-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:33.507397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:33.507363    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"
Oct 30 19:10:33.507613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:33.507388    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj" event={"ID":"5d684e4d-b989-4b39-b532-d5ac310147d8","Type":"ContainerDied","Data":"e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a"}
Oct 30 19:10:33.507613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:33.507456    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0bc29bf1ebb1d52fe1f58e4ed4c4d2b2d21b26693f2a5acf47351e2791fcf3a"
Oct 30 19:10:37.086996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.084405    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:10:37.090362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.089056    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5d684e4d-b989-4b39-b532-d5ac310147d8" containerName="revoke-keys"
Oct 30 19:10:37.090753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.090730    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d684e4d-b989-4b39-b532-d5ac310147d8" containerName="revoke-keys"
Oct 30 19:10:37.091438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.091396    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d684e4d-b989-4b39-b532-d5ac310147d8" containerName="revoke-keys"
Oct 30 19:10:37.097834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.097812    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.115624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.115590    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:10:37.127480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127450    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"routing-overlay\""
Oct 30 19:10:37.127691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127564    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-394aab6f23b5\""
Oct 30 19:10:37.127691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127632    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e94aa\""
Oct 30 19:10:37.127691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127637    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-e2e--0d1ddd5a-dockercfg-7twb9\""
Oct 30 19:10:37.127917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127721    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"kube-root-ca.crt\""
Oct 30 19:10:37.127917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127732    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"openshift-service-ca.crt\""
Oct 30 19:10:37.127917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.127817    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-ce5ac7999d53\""
Oct 30 19:10:37.174780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.174742    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.174939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.174876    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.175025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.175002    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcqz\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.175175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.175156    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.277205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.277174    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.277379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.277245    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.277379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.277315    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcqz\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.277532 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.277396    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.278144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.278077    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.278269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.278155    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.280065 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.280039    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.294445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.294422    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcqz\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.425306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.425221    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:37.725037 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:37.724988    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e88401_46d7_486d_ad7b_2524e1008c9f.slice/crio-88f17862705fb8de644ed8104452637ba1de957a63a2f43210965181f54f83be WatchSource:0}: Error finding container 88f17862705fb8de644ed8104452637ba1de957a63a2f43210965181f54f83be: Status 404 returned error can't find the container with id 88f17862705fb8de644ed8104452637ba1de957a63a2f43210965181f54f83be
Oct 30 19:10:37.728642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:37.728508    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:10:38.682495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:38.682459    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" event={"ID":"03e88401-46d7-486d-ad7b-2524e1008c9f","Type":"ContainerStarted","Data":"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"}
Oct 30 19:10:38.682998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:38.682519    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" event={"ID":"03e88401-46d7-486d-ad7b-2524e1008c9f","Type":"ContainerStarted","Data":"88f17862705fb8de644ed8104452637ba1de957a63a2f43210965181f54f83be"}
Oct 30 19:10:38.682998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:38.682573    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:38.706330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:38.706279    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" podStartSLOduration=1.706264912 podStartE2EDuration="1.706264912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:38.704037889 +0000 UTC m=+2258.967502860" watchObservedRunningTime="2026-10-30 19:10:38.706264912 +0000 UTC m=+2258.969729860"
Oct 30 19:10:42.969120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:42.968693    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:10:43.011348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.011320    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:10:43.059159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.023183    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.062371    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"payload-processing-external-model-e2e--f321fca4-dockercfg-z8tkg\""
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.062443    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"agc-cred-3b3535133443\""
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.064421    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"openshift-service-ca.crt\""
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.064513    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"kube-root-ca.crt\""
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.064571    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"routing-overlay\""
Oct 30 19:10:43.074117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.064618    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689c23\""
Oct 30 19:10:43.121605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.121560    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.121975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.121945    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.122169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.122134    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.323114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.274887    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bflv\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.424144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.380705    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.440106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.433324    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.440106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.433391    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.440106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.433529    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bflv\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.440106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.395382    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.440106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.434988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.560224 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.560191    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bflv\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.561260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.561232    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:43.804510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:43.804476    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:44.132308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:44.132279    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:10:44.893151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:44.890441    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" event={"ID":"ce87ce53-3994-4f05-a85e-ca01b0a7df31","Type":"ContainerStarted","Data":"6e015a496198acbbb349017d8eb88a7ebc961b74b8175dba3b8713b44463e920"}
Oct 30 19:10:44.893151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:44.890505    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" event={"ID":"ce87ce53-3994-4f05-a85e-ca01b0a7df31","Type":"ContainerStarted","Data":"024f6d1d28eb3581cc89eaed2e9b8ad0705427c58e0700392392acc6415739a4"}
Oct 30 19:10:44.893151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:44.891649    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:44.933258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:44.933209    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" podStartSLOduration=2.933195055 podStartE2EDuration="2.933195055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:44.912374409 +0000 UTC m=+2265.175839357" watchObservedRunningTime="2026-10-30 19:10:44.933195055 +0000 UTC m=+2265.196660004"
Oct 30 19:10:46.328861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.328789    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"]
Oct 30 19:10:46.335417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.335385    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.347139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.347076    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:46.347304 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.347257    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:46.351235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.351209    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"]
Oct 30 19:10:46.407883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.407858    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.408034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.407915    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.408232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.408204    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5l4z\" (UniqueName: \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.509354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.509327    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.509527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.509484    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5l4z\" (UniqueName: \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.509666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.509653    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.510412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.510389    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.512041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.512017    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.523559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.523524    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5l4z\" (UniqueName: \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\") pod \"maas-api-revoke-sub-03a333bd-j5q64\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.656383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.656315    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:46.957390 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:46.957361    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0f7e03_7145_4880_b0c4_a6da2909a5e1.slice/crio-1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc WatchSource:0}: Error finding container 1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc: Status 404 returned error can't find the container with id 1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc
Oct 30 19:10:46.957777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:46.957752    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"]
Oct 30 19:10:47.933108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:47.933062    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64" event={"ID":"1d0f7e03-7145-4880-b0c4-a6da2909a5e1","Type":"ContainerStarted","Data":"ede8e73bf29c290dc52f3c28549c33f112d62761cda4b77d60b37f5231df6361"}
Oct 30 19:10:47.933573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:47.933206    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64" event={"ID":"1d0f7e03-7145-4880-b0c4-a6da2909a5e1","Type":"ContainerStarted","Data":"1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc"}
Oct 30 19:10:47.955183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:47.955141    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64" podStartSLOduration=1.955130177 podStartE2EDuration="1.955130177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:47.953204718 +0000 UTC m=+2268.216669667" watchObservedRunningTime="2026-10-30 19:10:47.955130177 +0000 UTC m=+2268.218595125"
Oct 30 19:10:48.943866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:48.943826    5065 generic.go:356] "Generic (PLEG): container finished" podID="1d0f7e03-7145-4880-b0c4-a6da2909a5e1" containerID="ede8e73bf29c290dc52f3c28549c33f112d62761cda4b77d60b37f5231df6361" exitCode=0
Oct 30 19:10:48.944271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:48.943909    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64" event={"ID":"1d0f7e03-7145-4880-b0c4-a6da2909a5e1","Type":"ContainerDied","Data":"ede8e73bf29c290dc52f3c28549c33f112d62761cda4b77d60b37f5231df6361"}
Oct 30 19:10:49.722112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:49.718637    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:10:50.156439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.156414    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:50.169149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.169124    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\") pod \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") "
Oct 30 19:10:50.169293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.169216    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\" (UniqueName: \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\") pod \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") "
Oct 30 19:10:50.169355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.169296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\") pod \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\" (UID: \"1d0f7e03-7145-4880-b0c4-a6da2909a5e1\") "
Oct 30 19:10:50.169554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.169525    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca" pod "1d0f7e03-7145-4880-b0c4-a6da2909a5e1" (UID: "1d0f7e03-7145-4880-b0c4-a6da2909a5e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:50.169893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.169874    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:50.171587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.171561    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca" pod "1d0f7e03-7145-4880-b0c4-a6da2909a5e1" (UID: "1d0f7e03-7145-4880-b0c4-a6da2909a5e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:50.171673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.171610    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z" pod "1d0f7e03-7145-4880-b0c4-a6da2909a5e1" (UID: "1d0f7e03-7145-4880-b0c4-a6da2909a5e1"). InnerVolumeSpecName "kube-api-access-g5l4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:50.272049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.272019    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5l4z\" (UniqueName: \"kubernetes.io/projected/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-kube-api-access-g5l4z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:50.272049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.272050    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0f7e03-7145-4880-b0c4-a6da2909a5e1-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:50.965274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.965244    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"
Oct 30 19:10:50.965468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.965272    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64" event={"ID":"1d0f7e03-7145-4880-b0c4-a6da2909a5e1","Type":"ContainerDied","Data":"1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc"}
Oct 30 19:10:50.965468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:50.965322    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1134c3056675246b4c3d03a81dd7efcd8b03c0a5f7dcde62caa36161497be1dc"
Oct 30 19:10:53.075978 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:10:53.075942    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:10:54.017603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.017569    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:10:54.018480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.018462    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1d0f7e03-7145-4880-b0c4-a6da2909a5e1" containerName="revoke-keys"
Oct 30 19:10:54.018480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.018481    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0f7e03-7145-4880-b0c4-a6da2909a5e1" containerName="revoke-keys"
Oct 30 19:10:54.019160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.018654    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0f7e03-7145-4880-b0c4-a6da2909a5e1" containerName="revoke-keys"
Oct 30 19:10:54.022680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.022661    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.033452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.033432    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"agc-cred-56b60f463986\""
Oct 30 19:10:54.033606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.033581    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a717c7\""
Oct 30 19:10:54.033606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.033588    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-e2e--ebe4ecff-dockercfg-vjqtd\""
Oct 30 19:10:54.035804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.035785    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"routing-overlay\""
Oct 30 19:10:54.035804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.035793    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"kube-root-ca.crt\""
Oct 30 19:10:54.035973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.035812    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"openshift-service-ca.crt\""
Oct 30 19:10:54.043430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.043405    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:10:54.154120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.130131    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.154120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.130210    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.154120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.130510    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8rz\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.154120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.130589    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.310167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.307935    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8rz\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.310167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.308016    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.310167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.308261    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.310167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.308318    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.375111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.363018    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.400110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.375916    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.432111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.430767    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.458717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.449949    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8rz\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:54.639519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:54.639425    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:55.007225 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:55.007158    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7bfed_03fa_4c29_94cd_b1763c74f7c1.slice/crio-e6d056cfe89ed19527b1382ce4db1c62b3e077e194a5c8841ffede439bb6326c WatchSource:0}: Error finding container e6d056cfe89ed19527b1382ce4db1c62b3e077e194a5c8841ffede439bb6326c: Status 404 returned error can't find the container with id e6d056cfe89ed19527b1382ce4db1c62b3e077e194a5c8841ffede439bb6326c
Oct 30 19:10:55.007368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:55.007344    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:10:55.095818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:55.095784    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" event={"ID":"94e7bfed-03fa-4c29-94cd-b1763c74f7c1","Type":"ContainerStarted","Data":"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"}
Oct 30 19:10:55.095939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:55.095841    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" event={"ID":"94e7bfed-03fa-4c29-94cd-b1763c74f7c1","Type":"ContainerStarted","Data":"e6d056cfe89ed19527b1382ce4db1c62b3e077e194a5c8841ffede439bb6326c"}
Oct 30 19:10:55.096153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:55.096133    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:10:55.128879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:55.128827    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" podStartSLOduration=2.128813385 podStartE2EDuration="2.128813385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:55.127194932 +0000 UTC m=+2275.390659892" watchObservedRunningTime="2026-10-30 19:10:55.128813385 +0000 UTC m=+2275.392278333"
Oct 30 19:10:56.967119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:56.962526    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:10:58.923779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:58.923742    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"]
Oct 30 19:10:58.933874 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:58.933841    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:58.936722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:58.936700    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"]
Oct 30 19:10:58.940575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:58.940543    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:58.944290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:58.944266    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:10:59.020504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.020460    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzk49\" (UniqueName: \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.020666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.020593    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.020750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.020697    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.121907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.121871    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzk49\" (UniqueName: \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.122069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.121940    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.122069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.121995    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.122712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.122689    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.124735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.124706    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.135634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.135607    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzk49\" (UniqueName: \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\") pod \"maas-api-revoke-sub-0e304119-w6v8g\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.264339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.264310    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:10:59.568835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:10:59.568805    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"]
Oct 30 19:10:59.569604 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:10:59.569579    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2e1e73_2a6b_44a1_a538_a5ef8f6517b5.slice/crio-a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994 WatchSource:0}: Error finding container a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994: Status 404 returned error can't find the container with id a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994
Oct 30 19:11:00.161757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:00.161721    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g" event={"ID":"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5","Type":"ContainerStarted","Data":"3fa56b9727e27485b9c11d8065856bb82e15c42bd85833a7e53ca2373b3bbc6d"}
Oct 30 19:11:00.162215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:00.161786    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g" event={"ID":"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5","Type":"ContainerStarted","Data":"a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994"}
Oct 30 19:11:00.183879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:00.183833    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g" podStartSLOduration=2.183819062 podStartE2EDuration="2.183819062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:00.182234434 +0000 UTC m=+2280.445699399" watchObservedRunningTime="2026-10-30 19:11:00.183819062 +0000 UTC m=+2280.447284009"
Oct 30 19:11:01.173764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:01.173730    5065 generic.go:356] "Generic (PLEG): container finished" podID="ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" containerID="3fa56b9727e27485b9c11d8065856bb82e15c42bd85833a7e53ca2373b3bbc6d" exitCode=0
Oct 30 19:11:01.174279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:01.173813    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g" event={"ID":"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5","Type":"ContainerDied","Data":"3fa56b9727e27485b9c11d8065856bb82e15c42bd85833a7e53ca2373b3bbc6d"}
Oct 30 19:11:02.449205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.449173    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:11:02.532202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.532011    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\") pod \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") "
Oct 30 19:11:02.532514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.532491    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\" (UniqueName: \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\") pod \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") "
Oct 30 19:11:02.533060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.533004    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\") pod \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\" (UID: \"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5\") "
Oct 30 19:11:02.533597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.533570    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca" pod "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" (UID: "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:02.534590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.534551    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:02.535435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.535408    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca" pod "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" (UID: "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:02.535556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.535474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49" pod "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" (UID: "ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5"). InnerVolumeSpecName "kube-api-access-zzk49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:02.637030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.637000    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:02.637209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:02.637035    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzk49\" (UniqueName: \"kubernetes.io/projected/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5-kube-api-access-zzk49\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:03.242244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:03.242207    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"
Oct 30 19:11:03.242244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:03.242226    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g" event={"ID":"ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5","Type":"ContainerDied","Data":"a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994"}
Oct 30 19:11:03.242540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:03.242281    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7913178f8e41711d535e416c175500ecd40ebe144134ed2994d806318002994"
Oct 30 19:11:06.119553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:06.119528    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:11:07.475373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:07.475339    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:11:10.696198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.667281    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"]
Oct 30 19:11:10.696198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.668800    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" containerName="revoke-keys"
Oct 30 19:11:10.696198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.668825    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" containerName="revoke-keys"
Oct 30 19:11:10.696198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.669106    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" containerName="revoke-keys"
Oct 30 19:11:10.696198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.685991    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.699653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.699626    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:10.706773 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.706753    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:11:10.764285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.759213    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"]
Oct 30 19:11:10.837243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.837069    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.837473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.837319    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.837473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.837350    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8gr\" (UniqueName: \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.942380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.942342    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.968167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.955688    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.968478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.954811    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:10.968812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:10.968776    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8gr\" (UniqueName: \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:11.010431 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.001750    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:11.020140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.015327    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8gr\" (UniqueName: \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\") pod \"maas-api-revoke-sub-8290c0ce-qqqcz\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:11.035253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.035228    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:11.328898 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:11.328855    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b20f510_4181_418c_a499_4d1b061f9b43.slice/crio-fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246 WatchSource:0}: Error finding container fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246: Status 404 returned error can't find the container with id fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246
Oct 30 19:11:11.329854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.329799    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"]
Oct 30 19:11:11.390954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.390927    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz" event={"ID":"7b20f510-4181-418c-a499-4d1b061f9b43","Type":"ContainerStarted","Data":"fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246"}
Oct 30 19:11:11.662522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.662489    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"]
Oct 30 19:11:11.668841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.668821    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.682130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.680495    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"]
Oct 30 19:11:11.690766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.688753    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.690766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.688843    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5tm\" (UniqueName: \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.690766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.689079    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.828056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.827302    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.828056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.827402    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5tm\" (UniqueName: \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.828056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.827672    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.829997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.829965    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.850838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.841698    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.850995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.850931    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5tm\" (UniqueName: \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\") pod \"maas-api-revoke-sub-592b8461-rpns7\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:11.991634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:11.991598    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:12.297693 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:12.297662    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5920988_b0bf_4441_8ac6_f2e9a205e6dd.slice/crio-b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc WatchSource:0}: Error finding container b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc: Status 404 returned error can't find the container with id b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc
Oct 30 19:11:12.301588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.301437    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"]
Oct 30 19:11:12.406821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.406781    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7" event={"ID":"d5920988-b0bf-4441-8ac6-f2e9a205e6dd","Type":"ContainerStarted","Data":"b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc"}
Oct 30 19:11:12.409054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.409024    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz" event={"ID":"7b20f510-4181-418c-a499-4d1b061f9b43","Type":"ContainerStarted","Data":"ed10509787e5b70f6699f595223e10d1fa46d455a4e4953c86077582b8915fcc"}
Oct 30 19:11:12.444813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.444764    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz" podStartSLOduration=2.444750845 podStartE2EDuration="2.444750845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:12.440649428 +0000 UTC m=+2292.704114379" watchObservedRunningTime="2026-10-30 19:11:12.444750845 +0000 UTC m=+2292.708215792"
Oct 30 19:11:12.476499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.476465    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" podUID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" containerName="payload-processing" containerID="cri-o://b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56" gracePeriod=25
Oct 30 19:11:12.711474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.711441    5065 scope.go:122] "RemoveContainer" containerID="8d9ced282c4129a3d1c36f9c69a3ede99463e562ed9a86b6fa239d0c8f4c24eb"
Oct 30 19:11:12.734920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.734895    5065 scope.go:122] "RemoveContainer" containerID="bc689fe03482b2ab9dc87ced189dbfb9ca75b86f61d32771e6869e584ba872df"
Oct 30 19:11:12.746965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.746940    5065 scope.go:122] "RemoveContainer" containerID="541dd6303863c3f6a210389c66d9a8ed9f06087dc1f888ebc8bc7ea3be935364"
Oct 30 19:11:12.759595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.759571    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:11:12.760637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.760619    5065 scope.go:122] "RemoveContainer" containerID="5ceb015669e307514be91a9df55a47768c16005b919eba3ae34867b37114cef5"
Oct 30 19:11:12.772150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.772131    5065 scope.go:122] "RemoveContainer" containerID="816fc2b30a03adeac58b2cbeffa6a7540680c9ada262d2c08d8cc03591e5f769"
Oct 30 19:11:12.783785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.783762    5065 scope.go:122] "RemoveContainer" containerID="7b8637beb8444e5d5ede0c5b7e21f492dd8818b8b419bb0dfb98e7fd3e9ac288"
Oct 30 19:11:12.846786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.846761    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\") pod \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") "
Oct 30 19:11:12.847278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.846830    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\") pod \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") "
Oct 30 19:11:12.847278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.847021    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\") pod \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") "
Oct 30 19:11:12.847278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.847040    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\") pod \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\" (UID: \"94e7bfed-03fa-4c29-94cd-b1763c74f7c1\") "
Oct 30 19:11:12.851422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.851390    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume" pod "94e7bfed-03fa-4c29-94cd-b1763c74f7c1" (UID: "94e7bfed-03fa-4c29-94cd-b1763c74f7c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:12.851819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.851785    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume" pod "94e7bfed-03fa-4c29-94cd-b1763c74f7c1" (UID: "94e7bfed-03fa-4c29-94cd-b1763c74f7c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:12.852362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.852335    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz" pod "94e7bfed-03fa-4c29-94cd-b1763c74f7c1" (UID: "94e7bfed-03fa-4c29-94cd-b1763c74f7c1"). InnerVolumeSpecName "kube-api-access-tt8rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:12.854815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.854790    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials" pod "94e7bfed-03fa-4c29-94cd-b1763c74f7c1" (UID: "94e7bfed-03fa-4c29-94cd-b1763c74f7c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:13.040242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.989989    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:13.040242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.990043    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8rz\" (UniqueName: \"kubernetes.io/projected/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-kube-api-access-tt8rz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:13.040242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.990060    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:13.040242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:12.990073    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94e7bfed-03fa-4c29-94cd-b1763c74f7c1-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:13.423159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.423116    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7" event={"ID":"d5920988-b0bf-4441-8ac6-f2e9a205e6dd","Type":"ContainerStarted","Data":"234e980c0dc28b90f1c6d54256c08a60a7f45050cca783234c0f4bebfc1a4b70"}
Oct 30 19:11:13.432225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.432193    5065 generic.go:356] "Generic (PLEG): container finished" podID="7b20f510-4181-418c-a499-4d1b061f9b43" containerID="ed10509787e5b70f6699f595223e10d1fa46d455a4e4953c86077582b8915fcc" exitCode=0
Oct 30 19:11:13.432373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.432256    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz" event={"ID":"7b20f510-4181-418c-a499-4d1b061f9b43","Type":"ContainerDied","Data":"ed10509787e5b70f6699f595223e10d1fa46d455a4e4953c86077582b8915fcc"}
Oct 30 19:11:13.433651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.433629    5065 generic.go:356] "Generic (PLEG): container finished" podID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" containerID="b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56" exitCode=0
Oct 30 19:11:13.433767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.433674    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"
Oct 30 19:11:13.433767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.433707    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" event={"ID":"94e7bfed-03fa-4c29-94cd-b1763c74f7c1","Type":"ContainerDied","Data":"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"}
Oct 30 19:11:13.433767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.433759    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7" event={"ID":"94e7bfed-03fa-4c29-94cd-b1763c74f7c1","Type":"ContainerDied","Data":"e6d056cfe89ed19527b1382ce4db1c62b3e077e194a5c8841ffede439bb6326c"}
Oct 30 19:11:13.433913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.433797    5065 scope.go:122] "RemoveContainer" containerID="b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"
Oct 30 19:11:13.446478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.446444    5065 scope.go:122] "RemoveContainer" containerID="b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"
Oct 30 19:11:13.448399 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:13.448378    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56\": container with ID starting with b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56 not found: ID does not exist" containerID="b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"
Oct 30 19:11:13.448484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.448406    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56"} err="failed to get container status \"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56\": rpc error: code = NotFound desc = could not find container \"b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56\": container with ID starting with b9daca9d4c213e2d253002f826b823d7affc97c8cd65bbfd0c7438fde5abfe56 not found: ID does not exist"
Oct 30 19:11:13.449462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.449422    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7" podStartSLOduration=2.449410385 podStartE2EDuration="2.449410385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:13.446320187 +0000 UTC m=+2293.709785135" watchObservedRunningTime="2026-10-30 19:11:13.449410385 +0000 UTC m=+2293.712875332"
Oct 30 19:11:13.470864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.470844    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:11:13.480096 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.480059    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6mf6z7"]
Oct 30 19:11:13.937062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:13.937029    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" path="/var/lib/kubelet/pods/94e7bfed-03fa-4c29-94cd-b1763c74f7c1/volumes"
Oct 30 19:11:14.448723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.448691    5065 generic.go:356] "Generic (PLEG): container finished" podID="d5920988-b0bf-4441-8ac6-f2e9a205e6dd" containerID="234e980c0dc28b90f1c6d54256c08a60a7f45050cca783234c0f4bebfc1a4b70" exitCode=0
Oct 30 19:11:14.448925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.448780    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7" event={"ID":"d5920988-b0bf-4441-8ac6-f2e9a205e6dd","Type":"ContainerDied","Data":"234e980c0dc28b90f1c6d54256c08a60a7f45050cca783234c0f4bebfc1a4b70"}
Oct 30 19:11:14.688775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.688745    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:14.721023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.720941    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\") pod \"7b20f510-4181-418c-a499-4d1b061f9b43\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") "
Oct 30 19:11:14.721233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.721133    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\") pod \"7b20f510-4181-418c-a499-4d1b061f9b43\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") "
Oct 30 19:11:14.721303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.721273    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\" (UniqueName: \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\") pod \"7b20f510-4181-418c-a499-4d1b061f9b43\" (UID: \"7b20f510-4181-418c-a499-4d1b061f9b43\") "
Oct 30 19:11:14.721547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.721517    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca" pod "7b20f510-4181-418c-a499-4d1b061f9b43" (UID: "7b20f510-4181-418c-a499-4d1b061f9b43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:14.721927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.721909    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:14.723573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.723545    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca" pod "7b20f510-4181-418c-a499-4d1b061f9b43" (UID: "7b20f510-4181-418c-a499-4d1b061f9b43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:14.726734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.726710    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr" pod "7b20f510-4181-418c-a499-4d1b061f9b43" (UID: "7b20f510-4181-418c-a499-4d1b061f9b43"). InnerVolumeSpecName "kube-api-access-nr8gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:14.825748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.824245    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20f510-4181-418c-a499-4d1b061f9b43-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:14.825748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:14.824282    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8gr\" (UniqueName: \"kubernetes.io/projected/7b20f510-4181-418c-a499-4d1b061f9b43-kube-api-access-nr8gr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:15.465627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:15.465598    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"
Oct 30 19:11:15.475371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:15.469678    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz" event={"ID":"7b20f510-4181-418c-a499-4d1b061f9b43","Type":"ContainerDied","Data":"fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246"}
Oct 30 19:11:15.475371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:15.469733    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8ccace549b169ae3d418ec613d9a330e61519d2a69aba47164c4613477c246"
Oct 30 19:11:16.255125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.252601    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:16.357183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.340805    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\") pod \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") "
Oct 30 19:11:16.357183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.340900    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\") pod \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") "
Oct 30 19:11:16.357183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.340982    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\" (UniqueName: \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\") pod \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\" (UID: \"d5920988-b0bf-4441-8ac6-f2e9a205e6dd\") "
Oct 30 19:11:16.358048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.357997    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca" pod "d5920988-b0bf-4441-8ac6-f2e9a205e6dd" (UID: "d5920988-b0bf-4441-8ac6-f2e9a205e6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:16.430112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.413320    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:16.512207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.506161    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:11:16.621170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.609302    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm" pod "d5920988-b0bf-4441-8ac6-f2e9a205e6dd" (UID: "d5920988-b0bf-4441-8ac6-f2e9a205e6dd"). InnerVolumeSpecName "kube-api-access-hp5tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:16.629798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.624758    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca" pod "d5920988-b0bf-4441-8ac6-f2e9a205e6dd" (UID: "d5920988-b0bf-4441-8ac6-f2e9a205e6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:16.728163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.722793    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5tm\" (UniqueName: \"kubernetes.io/projected/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-kube-api-access-hp5tm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:16.824112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.798359    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5920988-b0bf-4441-8ac6-f2e9a205e6dd-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:16.824112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.805642    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7" event={"ID":"d5920988-b0bf-4441-8ac6-f2e9a205e6dd","Type":"ContainerDied","Data":"b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc"}
Oct 30 19:11:16.824112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.805711    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7d287b4568c793f5b429c70a9001e359bf90feb125f42ec1fd7fabfc978f3cc"
Oct 30 19:11:16.824112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.805798    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"
Oct 30 19:11:16.872938 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.872836    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:11:16.892112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:16.873211    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:17.121113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:17.100938    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:17.155285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:17.132511    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:17.155285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:17.132998    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 19:11:17.239112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:17.214600    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:17.404114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:17.382443    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:18.421786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.421755    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"]
Oct 30 19:11:18.422654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422638    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7b20f510-4181-418c-a499-4d1b061f9b43" containerName="revoke-keys"
Oct 30 19:11:18.422717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422656    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b20f510-4181-418c-a499-4d1b061f9b43" containerName="revoke-keys"
Oct 30 19:11:18.422717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422673    5065 cpu_manager.go:414] "containerMap: removing container" podUID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" containerName="payload-processing"
Oct 30 19:11:18.422717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422677    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" containerName="payload-processing"
Oct 30 19:11:18.422717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422685    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d5920988-b0bf-4441-8ac6-f2e9a205e6dd" containerName="revoke-keys"
Oct 30 19:11:18.422717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422689    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5920988-b0bf-4441-8ac6-f2e9a205e6dd" containerName="revoke-keys"
Oct 30 19:11:18.422892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422878    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b20f510-4181-418c-a499-4d1b061f9b43" containerName="revoke-keys"
Oct 30 19:11:18.422892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422890    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e7bfed-03fa-4c29-94cd-b1763c74f7c1" containerName="payload-processing"
Oct 30 19:11:18.422962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.422897    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5920988-b0bf-4441-8ac6-f2e9a205e6dd" containerName="revoke-keys"
Oct 30 19:11:18.427056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.427039    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.433978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.433720    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:18.433978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.433906    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:11:18.437912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.437760    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"]
Oct 30 19:11:18.514140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.514102    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.514301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.514194    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r695n\" (UniqueName: \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.514413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.514395    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.615628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.615590    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.615795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.615632    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r695n\" (UniqueName: \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.615795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.615783    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.616348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.616328    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.618376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.618353    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.629154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.629135    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r695n\" (UniqueName: \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\") pod \"maas-api-revoke-sub-c159b54e-cm5zt\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:18.759392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:18.759364    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:19.171398 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:19.171369    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a679010_ea3d_437c_96c4_e79f57cadecc.slice/crio-0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e WatchSource:0}: Error finding container 0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e: Status 404 returned error can't find the container with id 0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e
Oct 30 19:11:19.172300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:19.172274    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"]
Oct 30 19:11:19.869937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:19.869852    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt" event={"ID":"2a679010-ea3d-437c-96c4-e79f57cadecc","Type":"ContainerStarted","Data":"b10ea5bcd2ccaca4a9d1ddba212dbb84fcd7daf99a1f27754ccdc6f9a8e10599"}
Oct 30 19:11:19.869937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:19.869903    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt" event={"ID":"2a679010-ea3d-437c-96c4-e79f57cadecc","Type":"ContainerStarted","Data":"0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e"}
Oct 30 19:11:19.910964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:19.910905    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt" podStartSLOduration=1.9108876989999999 podStartE2EDuration="1.910887699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:19.906783532 +0000 UTC m=+2300.170248494" watchObservedRunningTime="2026-10-30 19:11:19.910887699 +0000 UTC m=+2300.174352648"
Oct 30 19:11:20.882012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:20.881970    5065 generic.go:356] "Generic (PLEG): container finished" podID="2a679010-ea3d-437c-96c4-e79f57cadecc" containerID="b10ea5bcd2ccaca4a9d1ddba212dbb84fcd7daf99a1f27754ccdc6f9a8e10599" exitCode=0
Oct 30 19:11:20.882496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:20.882127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt" event={"ID":"2a679010-ea3d-437c-96c4-e79f57cadecc","Type":"ContainerDied","Data":"b10ea5bcd2ccaca4a9d1ddba212dbb84fcd7daf99a1f27754ccdc6f9a8e10599"}
Oct 30 19:11:21.507335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.507277    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" podUID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" containerName="payload-processing" containerID="cri-o://581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6" gracePeriod=25
Oct 30 19:11:21.868945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.868911    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:11:21.884775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.884744    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:11:21.924223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.923309    5065 generic.go:356] "Generic (PLEG): container finished" podID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" containerID="581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6" exitCode=0
Oct 30 19:11:21.924223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.923390    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"
Oct 30 19:11:21.925270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.925236    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" event={"ID":"fb073d66-ef5f-43ec-a96b-f1c664810e7e","Type":"ContainerDied","Data":"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"}
Oct 30 19:11:21.925391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.925345    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth" event={"ID":"fb073d66-ef5f-43ec-a96b-f1c664810e7e","Type":"ContainerDied","Data":"de81dc524da6d12f9909770867cec20714b838b53994a0b0b65008b71ce3df57"}
Oct 30 19:11:21.925444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.925402    5065 scope.go:122] "RemoveContainer" containerID="581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"
Oct 30 19:11:21.949342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.947671    5065 scope.go:122] "RemoveContainer" containerID="581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"
Oct 30 19:11:21.949342 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:21.948019    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6\": container with ID starting with 581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6 not found: ID does not exist" containerID="581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"
Oct 30 19:11:21.949342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:21.948059    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6"} err="failed to get container status \"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6\": rpc error: code = NotFound desc = could not find container \"581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6\": container with ID starting with 581310cdd7b2d0426e8513892463892326333e5833d756aa33d32801c0a68ef6 not found: ID does not exist"
Oct 30 19:11:22.013822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.013776    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\") pod \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") "
Oct 30 19:11:22.014168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.014144    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\") pod \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") "
Oct 30 19:11:22.014328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.014311    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\") pod \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") "
Oct 30 19:11:22.014417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.014393    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\") pod \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\" (UID: \"fb073d66-ef5f-43ec-a96b-f1c664810e7e\") "
Oct 30 19:11:22.015684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.014578    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume" pod "fb073d66-ef5f-43ec-a96b-f1c664810e7e" (UID: "fb073d66-ef5f-43ec-a96b-f1c664810e7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:22.015684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.015309    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume" pod "fb073d66-ef5f-43ec-a96b-f1c664810e7e" (UID: "fb073d66-ef5f-43ec-a96b-f1c664810e7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:22.016587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.016451    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.016587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.016476    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb073d66-ef5f-43ec-a96b-f1c664810e7e-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.019875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.019849    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials" pod "fb073d66-ef5f-43ec-a96b-f1c664810e7e" (UID: "fb073d66-ef5f-43ec-a96b-f1c664810e7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:22.025228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.025205    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb" pod "fb073d66-ef5f-43ec-a96b-f1c664810e7e" (UID: "fb073d66-ef5f-43ec-a96b-f1c664810e7e"). InnerVolumeSpecName "kube-api-access-pqhmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:22.131035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.120323    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.131035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.120366    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhmb\" (UniqueName: \"kubernetes.io/projected/fb073d66-ef5f-43ec-a96b-f1c664810e7e-kube-api-access-pqhmb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.535127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.532347    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:11:22.545935 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.545900    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7b74ngvth"]
Oct 30 19:11:22.586824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.586792    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:22.646837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.646802    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\") pod \"2a679010-ea3d-437c-96c4-e79f57cadecc\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") "
Oct 30 19:11:22.647201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.647175    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\" (UniqueName: \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\") pod \"2a679010-ea3d-437c-96c4-e79f57cadecc\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") "
Oct 30 19:11:22.647341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.647305    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\") pod \"2a679010-ea3d-437c-96c4-e79f57cadecc\" (UID: \"2a679010-ea3d-437c-96c4-e79f57cadecc\") "
Oct 30 19:11:22.647905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.647866    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca" pod "2a679010-ea3d-437c-96c4-e79f57cadecc" (UID: "2a679010-ea3d-437c-96c4-e79f57cadecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:22.648969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.648941    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.649688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.649666    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n" pod "2a679010-ea3d-437c-96c4-e79f57cadecc" (UID: "2a679010-ea3d-437c-96c4-e79f57cadecc"). InnerVolumeSpecName "kube-api-access-r695n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:22.654945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.654921    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca" pod "2a679010-ea3d-437c-96c4-e79f57cadecc" (UID: "2a679010-ea3d-437c-96c4-e79f57cadecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:22.762166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.757499    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a679010-ea3d-437c-96c4-e79f57cadecc-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.762166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.757532    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r695n\" (UniqueName: \"kubernetes.io/projected/2a679010-ea3d-437c-96c4-e79f57cadecc-kube-api-access-r695n\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:22.936993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.936958    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"
Oct 30 19:11:22.936993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.936979    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt" event={"ID":"2a679010-ea3d-437c-96c4-e79f57cadecc","Type":"ContainerDied","Data":"0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e"}
Oct 30 19:11:22.937499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:22.937032    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6486b5105e44fec0d4ed422cf6c524a6b7bf28f3c765620e2527c674fec40e"
Oct 30 19:11:23.062645 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:23.061620    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:11:23.894761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:23.894722    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" path="/var/lib/kubelet/pods/fb073d66-ef5f-43ec-a96b-f1c664810e7e/volumes"
Oct 30 19:11:24.565620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.565589    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"]
Oct 30 19:11:24.566536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566517    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2a679010-ea3d-437c-96c4-e79f57cadecc" containerName="revoke-keys"
Oct 30 19:11:24.566633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566537    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a679010-ea3d-437c-96c4-e79f57cadecc" containerName="revoke-keys"
Oct 30 19:11:24.566633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566591    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" containerName="payload-processing"
Oct 30 19:11:24.566633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566596    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" containerName="payload-processing"
Oct 30 19:11:24.566750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566741    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb073d66-ef5f-43ec-a96b-f1c664810e7e" containerName="payload-processing"
Oct 30 19:11:24.566792 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.566758    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a679010-ea3d-437c-96c4-e79f57cadecc" containerName="revoke-keys"
Oct 30 19:11:24.570875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.570854    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.595121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.579224    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:24.595121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.579263    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:11:24.627545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.627424    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"]
Oct 30 19:11:24.741758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.741725    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.741940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.741861    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.741940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.741937    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.843471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.843393    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.843714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.843494    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.843793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.843759    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.844324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.844297    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.846658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.846636    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.857929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.857792    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\") pod \"maas-api-revoke-sub-d89265e2-z56qp\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:24.958234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:24.958204    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:25.266916 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:25.266859    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cf3c3e_4db1_4bec_9088_7ae6cdc3b5ff.slice/crio-43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add WatchSource:0}: Error finding container 43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add: Status 404 returned error can't find the container with id 43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add
Oct 30 19:11:25.273995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:25.273967    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"]
Oct 30 19:11:25.967316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:25.967238    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp" event={"ID":"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff","Type":"ContainerStarted","Data":"6ac96c7ff3d6e75392bd1796671581c926d62db6f217e7113239601b933eae04"}
Oct 30 19:11:25.967316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:25.967287    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp" event={"ID":"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff","Type":"ContainerStarted","Data":"43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add"}
Oct 30 19:11:25.990710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:25.990660    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp" podStartSLOduration=1.990646785 podStartE2EDuration="1.990646785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:25.989323994 +0000 UTC m=+2306.252788954" watchObservedRunningTime="2026-10-30 19:11:25.990646785 +0000 UTC m=+2306.254111732"
Oct 30 19:11:26.870009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:26.869968    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" podUID="03e88401-46d7-486d-ad7b-2524e1008c9f" containerName="payload-processing" containerID="cri-o://66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244" gracePeriod=25
Oct 30 19:11:26.979148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:26.979042    5065 generic.go:356] "Generic (PLEG): container finished" podID="25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" containerID="6ac96c7ff3d6e75392bd1796671581c926d62db6f217e7113239601b933eae04" exitCode=0
Oct 30 19:11:26.979607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:26.979160    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp" event={"ID":"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff","Type":"ContainerDied","Data":"6ac96c7ff3d6e75392bd1796671581c926d62db6f217e7113239601b933eae04"}
Oct 30 19:11:27.322431 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.322406    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:11:27.409527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.409498    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\") pod \"03e88401-46d7-486d-ad7b-2524e1008c9f\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") "
Oct 30 19:11:27.409733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.409536    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\") pod \"03e88401-46d7-486d-ad7b-2524e1008c9f\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") "
Oct 30 19:11:27.409733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.409605    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\") pod \"03e88401-46d7-486d-ad7b-2524e1008c9f\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") "
Oct 30 19:11:27.409733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.409681    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\") pod \"03e88401-46d7-486d-ad7b-2524e1008c9f\" (UID: \"03e88401-46d7-486d-ad7b-2524e1008c9f\") "
Oct 30 19:11:27.410195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.410164    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume" pod "03e88401-46d7-486d-ad7b-2524e1008c9f" (UID: "03e88401-46d7-486d-ad7b-2524e1008c9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:27.410195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.410222    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume" pod "03e88401-46d7-486d-ad7b-2524e1008c9f" (UID: "03e88401-46d7-486d-ad7b-2524e1008c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:27.410563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.410544    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.410563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.410569    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e88401-46d7-486d-ad7b-2524e1008c9f-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.412294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.412269    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials" pod "03e88401-46d7-486d-ad7b-2524e1008c9f" (UID: "03e88401-46d7-486d-ad7b-2524e1008c9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:27.412975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.412948    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz" pod "03e88401-46d7-486d-ad7b-2524e1008c9f" (UID: "03e88401-46d7-486d-ad7b-2524e1008c9f"). InnerVolumeSpecName "kube-api-access-qrcqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:27.512605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.512575    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.512605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:27.512605    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrcqz\" (UniqueName: \"kubernetes.io/projected/03e88401-46d7-486d-ad7b-2524e1008c9f-kube-api-access-qrcqz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:28.041126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.038420    5065 generic.go:356] "Generic (PLEG): container finished" podID="03e88401-46d7-486d-ad7b-2524e1008c9f" containerID="66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244" exitCode=0
Oct 30 19:11:28.041126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.038976    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" event={"ID":"03e88401-46d7-486d-ad7b-2524e1008c9f","Type":"ContainerDied","Data":"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"}
Oct 30 19:11:28.041126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.039013    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"
Oct 30 19:11:28.041126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.040107    5065 scope.go:122] "RemoveContainer" containerID="66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"
Oct 30 19:11:28.041126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.039023    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v" event={"ID":"03e88401-46d7-486d-ad7b-2524e1008c9f","Type":"ContainerDied","Data":"88f17862705fb8de644ed8104452637ba1de957a63a2f43210965181f54f83be"}
Oct 30 19:11:28.058781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.058752    5065 scope.go:122] "RemoveContainer" containerID="66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"
Oct 30 19:11:28.059175 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:28.059135    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244\": container with ID starting with 66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244 not found: ID does not exist" containerID="66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"
Oct 30 19:11:28.059326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.059187    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244"} err="failed to get container status \"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244\": rpc error: code = NotFound desc = could not find container \"66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244\": container with ID starting with 66541ab09203fdc9d99fb05580a4f4b5d8cf687463f3c7c1a18ec65eaac25244 not found: ID does not exist"
Oct 30 19:11:28.075275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.075249    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:11:28.080817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.080788    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-5cbdmbt6v"]
Oct 30 19:11:28.275375 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.275351    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:28.339997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.339963    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\") pod \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") "
Oct 30 19:11:28.340204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.340187    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\") pod \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") "
Oct 30 19:11:28.340279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.340265    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\") pod \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\" (UID: \"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff\") "
Oct 30 19:11:28.340351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.340282    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca" pod "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" (UID: "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:28.340987 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.340887    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:28.342469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.342441    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca" pod "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" (UID: "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:28.342684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.342661    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm" pod "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" (UID: "25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff"). InnerVolumeSpecName "kube-api-access-zg2wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:28.442001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.441973    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-kube-api-access-zg2wm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:28.442001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:28.441999    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:29.050386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:29.050359    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"
Oct 30 19:11:29.050887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:29.050393    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp" event={"ID":"25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff","Type":"ContainerDied","Data":"43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add"}
Oct 30 19:11:29.050887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:29.050439    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c829d711f857e51d441eac9335326ccffda69e969f5b2df914f24fc5619add"
Oct 30 19:11:29.612341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:29.611583    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:11:30.031071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:30.031034    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e88401-46d7-486d-ad7b-2524e1008c9f" path="/var/lib/kubelet/pods/03e88401-46d7-486d-ad7b-2524e1008c9f/volumes"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.915532    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.917605    5065 cpu_manager.go:414] "containerMap: removing container" podUID="03e88401-46d7-486d-ad7b-2524e1008c9f" containerName="payload-processing"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.917657    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e88401-46d7-486d-ad7b-2524e1008c9f" containerName="payload-processing"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.917814    5065 cpu_manager.go:414] "containerMap: removing container" podUID="25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" containerName="revoke-keys"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.917823    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" containerName="revoke-keys"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.918399    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e88401-46d7-486d-ad7b-2524e1008c9f" containerName="payload-processing"
Oct 30 19:11:32.920111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.918443    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" containerName="revoke-keys"
Oct 30 19:11:32.947569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.947535    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:32.947742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.947659    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:32.957445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.957413    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-283e3f5c73ea\""
Oct 30 19:11:32.957612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:32.957549    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-e2e--0575c797-dockercfg-4s2r8\""
Oct 30 19:11:33.021925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.021849    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.021925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.021918    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.022361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.022224    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.022361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.022251    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krt6m\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.123454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.123421    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.123731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.123699    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.123998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.123980    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.124067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.124007    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krt6m\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.124903 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.124869    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.126754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.126729    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.126857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.126751    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.137151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.137128    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krt6m\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.276714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.275412    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:33.855776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.855701    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:11:33.871009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.869790    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:11:33.871009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.869890    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:33.875220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.875180    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-ff55e368d98a\""
Oct 30 19:11:33.969603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.969570    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:33.969957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.969865    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwrg\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:33.970039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.970014    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:33.970216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.970195    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:33.978622 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:33.978595    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd5bc69_6cf7_47e9_8338_5fd12da9da13.slice/crio-b100fe77e0d73cc98941cf7a96feacfc62f86038a5d92fbb8289e3cd93c9dbea WatchSource:0}: Error finding container b100fe77e0d73cc98941cf7a96feacfc62f86038a5d92fbb8289e3cd93c9dbea: Status 404 returned error can't find the container with id b100fe77e0d73cc98941cf7a96feacfc62f86038a5d92fbb8289e3cd93c9dbea
Oct 30 19:11:33.979753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:33.979728    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:34.071709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.071668    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwrg\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.071840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.071737    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.071840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.071792    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.071966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.071867    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.072573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.072505    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.072751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.072667    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.074448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.074409    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.085308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.085285    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwrg\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.214257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.213469    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:34.243780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.243719    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" event={"ID":"bdd5bc69-6cf7-47e9-8338-5fd12da9da13","Type":"ContainerStarted","Data":"ace94dc4a3226a299e59d4e27eb290b9969b9e75efe34c4ed44a00d0d6366e09"}
Oct 30 19:11:34.243919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.243816    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" event={"ID":"bdd5bc69-6cf7-47e9-8338-5fd12da9da13","Type":"ContainerStarted","Data":"b100fe77e0d73cc98941cf7a96feacfc62f86038a5d92fbb8289e3cd93c9dbea"}
Oct 30 19:11:34.245135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.245111    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:34.271396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.271324    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" podStartSLOduration=2.271305485 podStartE2EDuration="2.271305485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:34.266130751 +0000 UTC m=+2314.529595715" watchObservedRunningTime="2026-10-30 19:11:34.271305485 +0000 UTC m=+2314.534770433"
Oct 30 19:11:34.530695 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:34.530664    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b273358_1820_4ab4_b065_5b07aab2b98b.slice/crio-fee458c4e948b8358964d6ec54e0f8ede15f4a64ce7d27e753bd9176199f8145 WatchSource:0}: Error finding container fee458c4e948b8358964d6ec54e0f8ede15f4a64ce7d27e753bd9176199f8145: Status 404 returned error can't find the container with id fee458c4e948b8358964d6ec54e0f8ede15f4a64ce7d27e753bd9176199f8145
Oct 30 19:11:34.531385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.531359    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:11:34.612474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.612444    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" podUID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" containerName="payload-processing" containerID="cri-o://fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db" gracePeriod=25
Oct 30 19:11:34.903120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.897485    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:11:34.995479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.994235    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\") pod \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") "
Oct 30 19:11:34.995479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.994441    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\") pod \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") "
Oct 30 19:11:34.995479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.994674    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\") pod \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") "
Oct 30 19:11:34.995479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.994732    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\") pod \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\" (UID: \"d108de79-a3c8-4bb8-a687-692d9e8e7bc1\") "
Oct 30 19:11:34.997490 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.997239    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume" pod "d108de79-a3c8-4bb8-a687-692d9e8e7bc1" (UID: "d108de79-a3c8-4bb8-a687-692d9e8e7bc1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:34.997850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:34.997792    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume" pod "d108de79-a3c8-4bb8-a687-692d9e8e7bc1" (UID: "d108de79-a3c8-4bb8-a687-692d9e8e7bc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:35.014996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.013587    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials" pod "d108de79-a3c8-4bb8-a687-692d9e8e7bc1" (UID: "d108de79-a3c8-4bb8-a687-692d9e8e7bc1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:35.015741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.015561    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v" pod "d108de79-a3c8-4bb8-a687-692d9e8e7bc1" (UID: "d108de79-a3c8-4bb8-a687-692d9e8e7bc1"). InnerVolumeSpecName "kube-api-access-wvq2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:35.116198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.105822    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvq2v\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-kube-api-access-wvq2v\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:35.116198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.111188    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:35.116198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.111220    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:35.116198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.111234    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d108de79-a3c8-4bb8-a687-692d9e8e7bc1-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:35.253775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.253739    5065 generic.go:356] "Generic (PLEG): container finished" podID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" containerID="fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db" exitCode=0
Oct 30 19:11:35.254540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.253792    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"
Oct 30 19:11:35.254540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.253824    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" event={"ID":"d108de79-a3c8-4bb8-a687-692d9e8e7bc1","Type":"ContainerDied","Data":"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"}
Oct 30 19:11:35.254540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.253886    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk" event={"ID":"d108de79-a3c8-4bb8-a687-692d9e8e7bc1","Type":"ContainerDied","Data":"d6a0ff89b94a19a199a2d05e54655fdd7be1439e96c80fd5532867a040ccda73"}
Oct 30 19:11:35.254540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.253911    5065 scope.go:122] "RemoveContainer" containerID="fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"
Oct 30 19:11:35.256361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.256332    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" event={"ID":"9b273358-1820-4ab4-b065-5b07aab2b98b","Type":"ContainerStarted","Data":"5eb287eebee2971f43e4a83010efadbdd34714fea8149657cf9d26d4b4ad2a77"}
Oct 30 19:11:35.256489 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.256390    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" event={"ID":"9b273358-1820-4ab4-b065-5b07aab2b98b","Type":"ContainerStarted","Data":"fee458c4e948b8358964d6ec54e0f8ede15f4a64ce7d27e753bd9176199f8145"}
Oct 30 19:11:35.256552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.256478    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:35.266491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.266475    5065 scope.go:122] "RemoveContainer" containerID="fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"
Oct 30 19:11:35.266747 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:35.266729    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db\": container with ID starting with fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db not found: ID does not exist" containerID="fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"
Oct 30 19:11:35.266816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.266758    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db"} err="failed to get container status \"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db\": rpc error: code = NotFound desc = could not find container \"fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db\": container with ID starting with fe33b5346ab7e9afa28983c5ce96c54e8d1e90c8cf297dee780272e685dc84db not found: ID does not exist"
Oct 30 19:11:35.300964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.300921    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" podStartSLOduration=2.300908342 podStartE2EDuration="2.300908342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:35.281520111 +0000 UTC m=+2315.544985105" watchObservedRunningTime="2026-10-30 19:11:35.300908342 +0000 UTC m=+2315.564373291"
Oct 30 19:11:35.316010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.315962    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:11:35.321585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.321559    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-749bw5zmk"]
Oct 30 19:11:35.863109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.860830    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:35.863109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.862363    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" containerName="payload-processing"
Oct 30 19:11:35.863109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.862384    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" containerName="payload-processing"
Oct 30 19:11:35.863109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.862717    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" containerName="payload-processing"
Oct 30 19:11:35.906109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.901240    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:35.917456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:35.915360    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-54fc64a67d3d\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.038697    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-bda27d1a8117\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.038778    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.038873    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.038937    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.039018    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-hq7ks\""
Oct 30 19:11:36.043416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.039106    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:11:36.084195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.067271    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ttj\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.084195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.067440    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.084195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.067510    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.084195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.067560    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.218181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.174031    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d108de79-a3c8-4bb8-a687-692d9e8e7bc1" path="/var/lib/kubelet/pods/d108de79-a3c8-4bb8-a687-692d9e8e7bc1/volumes"
Oct 30 19:11:36.219182 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.219145    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ttj\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.235114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.223328    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.246119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.235845    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.341529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.320492    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:36.341529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.237113    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.341529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.332670    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.342509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.342325    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.349102 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.343063    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.361114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.360818    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ttj\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.463997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.463964    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"]
Oct 30 19:11:36.469409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.469352    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.476446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.476422    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:36.476620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.476518    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:11:36.480025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.480003    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"]
Oct 30 19:11:36.585115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.577327    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pk9g\" (UniqueName: \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.585115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.577444    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.585115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.577637    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.657112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.638810    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:36.721595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.721508    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk9g\" (UniqueName: \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.724627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.724596    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.725697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.725672    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.730108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.725792    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.744499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.744474    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.774814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.774785    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pk9g\" (UniqueName: \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\") pod \"maas-api-revoke-sub-46f68a25-88227\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:36.798123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:36.790811    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:37.261008 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:37.260977    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bff3ee0_8330_4f87_afa4_36c1412583a2.slice/crio-0fe5b34149e152c08992074cb0d8f58342d1aa2084139ad86571d292b5f3d673 WatchSource:0}: Error finding container 0fe5b34149e152c08992074cb0d8f58342d1aa2084139ad86571d292b5f3d673: Status 404 returned error can't find the container with id 0fe5b34149e152c08992074cb0d8f58342d1aa2084139ad86571d292b5f3d673
Oct 30 19:11:37.264016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:37.263994    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:37.296941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:37.296892    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"]
Oct 30 19:11:37.409620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:37.409542    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" event={"ID":"1bff3ee0-8330-4f87-afa4-36c1412583a2","Type":"ContainerStarted","Data":"0fe5b34149e152c08992074cb0d8f58342d1aa2084139ad86571d292b5f3d673"}
Oct 30 19:11:37.412431 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:37.412403    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227" event={"ID":"8c7e09ea-3065-4d25-be85-8960f1075956","Type":"ContainerStarted","Data":"8288b6772431607e3cd7d35552057a4b8a7100e6669d546e26f5a5dd72695449"}
Oct 30 19:11:38.427708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:38.427654    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" event={"ID":"1bff3ee0-8330-4f87-afa4-36c1412583a2","Type":"ContainerStarted","Data":"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"}
Oct 30 19:11:38.428249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:38.427950    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:38.429794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:38.429757    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227" event={"ID":"8c7e09ea-3065-4d25-be85-8960f1075956","Type":"ContainerStarted","Data":"97b033979a4dfaae6087aa3e196ad5c94a5cc0c36309d987546698e1202c50cd"}
Oct 30 19:11:38.542118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:38.538111    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" podStartSLOduration=3.538071854 podStartE2EDuration="3.538071854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:38.475049201 +0000 UTC m=+2318.738514150" watchObservedRunningTime="2026-10-30 19:11:38.538071854 +0000 UTC m=+2318.801536804"
Oct 30 19:11:38.586108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:38.580698    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227" podStartSLOduration=2.580680196 podStartE2EDuration="2.580680196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:38.542523503 +0000 UTC m=+2318.805988452" watchObservedRunningTime="2026-10-30 19:11:38.580680196 +0000 UTC m=+2318.844145145"
Oct 30 19:11:39.442665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.442633    5065 generic.go:356] "Generic (PLEG): container finished" podID="8c7e09ea-3065-4d25-be85-8960f1075956" containerID="97b033979a4dfaae6087aa3e196ad5c94a5cc0c36309d987546698e1202c50cd" exitCode=0
Oct 30 19:11:39.443253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.442719    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227" event={"ID":"8c7e09ea-3065-4d25-be85-8960f1075956","Type":"ContainerDied","Data":"97b033979a4dfaae6087aa3e196ad5c94a5cc0c36309d987546698e1202c50cd"}
Oct 30 19:11:39.622974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.622942    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:11:39.628683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.628663    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.635613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.635587    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"kube-root-ca.crt\""
Oct 30 19:11:39.635754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.635687    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-e2e--307b9dfd-dockercfg-q6kdn\""
Oct 30 19:11:39.635822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.635757    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"openshift-service-ca.crt\""
Oct 30 19:11:39.635822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.635792    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8c554\""
Oct 30 19:11:39.635937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.635857    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"agc-cred-ffbcd05a91cf\""
Oct 30 19:11:39.637864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.637827    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"routing-overlay\""
Oct 30 19:11:39.641857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.641837    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:11:39.782109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.781390    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.782109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.781452    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7dr\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.782109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.781651    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.782109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.781747    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.889108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.888407    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.889108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.888678    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.889108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.888717    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7dr\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.889108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.888923    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.905106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.889901    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.928109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.907899    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.928109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.908963    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.951106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.949813    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7dr\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:39.951106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:39.950129    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:40.566360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.562314    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:11:40.573813 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:40.573780    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b205bc_a004_4928_8d8c_47888b37d898.slice/crio-ad97abfa4b7bb3ca1dedaa5a5d40d249e032affeafa39ddf6d0d6a5bf669017c WatchSource:0}: Error finding container ad97abfa4b7bb3ca1dedaa5a5d40d249e032affeafa39ddf6d0d6a5bf669017c: Status 404 returned error can't find the container with id ad97abfa4b7bb3ca1dedaa5a5d40d249e032affeafa39ddf6d0d6a5bf669017c
Oct 30 19:11:40.802905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.802871    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:40.926900 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.926866    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\") pod \"8c7e09ea-3065-4d25-be85-8960f1075956\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") "
Oct 30 19:11:40.927730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.927703    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\" (UniqueName: \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\") pod \"8c7e09ea-3065-4d25-be85-8960f1075956\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") "
Oct 30 19:11:40.928388 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.928238    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca" pod "8c7e09ea-3065-4d25-be85-8960f1075956" (UID: "8c7e09ea-3065-4d25-be85-8960f1075956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:40.932376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.929926    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca" pod "8c7e09ea-3065-4d25-be85-8960f1075956" (UID: "8c7e09ea-3065-4d25-be85-8960f1075956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:40.932376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.931062    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\") pod \"8c7e09ea-3065-4d25-be85-8960f1075956\" (UID: \"8c7e09ea-3065-4d25-be85-8960f1075956\") "
Oct 30 19:11:40.932376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.931987    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:40.932376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.932010    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e09ea-3065-4d25-be85-8960f1075956-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:40.932376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:40.932112    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g" pod "8c7e09ea-3065-4d25-be85-8960f1075956" (UID: "8c7e09ea-3065-4d25-be85-8960f1075956"). InnerVolumeSpecName "kube-api-access-9pk9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:41.052117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.041142    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pk9g\" (UniqueName: \"kubernetes.io/projected/8c7e09ea-3065-4d25-be85-8960f1075956-kube-api-access-9pk9g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:41.826359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.822758    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227" event={"ID":"8c7e09ea-3065-4d25-be85-8960f1075956","Type":"ContainerDied","Data":"8288b6772431607e3cd7d35552057a4b8a7100e6669d546e26f5a5dd72695449"}
Oct 30 19:11:41.826359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.822828    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8288b6772431607e3cd7d35552057a4b8a7100e6669d546e26f5a5dd72695449"
Oct 30 19:11:41.826359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.822907    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"
Oct 30 19:11:41.844786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.844748    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" event={"ID":"61b205bc-a004-4928-8d8c-47888b37d898","Type":"ContainerStarted","Data":"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"}
Oct 30 19:11:41.844960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.844816    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" event={"ID":"61b205bc-a004-4928-8d8c-47888b37d898","Type":"ContainerStarted","Data":"ad97abfa4b7bb3ca1dedaa5a5d40d249e032affeafa39ddf6d0d6a5bf669017c"}
Oct 30 19:11:41.846201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.846115    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:41.926140 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:41.924987    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:41.926140 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:41.925020    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:41.926140 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:41.925112    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:42.425070562 +0000 UTC m=+2322.688535502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:41.933521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:41.930782    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" podStartSLOduration=2.930762404 podStartE2EDuration="2.930762404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:41.894766117 +0000 UTC m=+2322.158231078" watchObservedRunningTime="2026-10-30 19:11:41.930762404 +0000 UTC m=+2322.194227353"
Oct 30 19:11:42.442952 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:42.442913    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:42.443224 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:42.443208    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:42.443396 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:42.443385    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:43.443364144 +0000 UTC m=+2323.706829089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:43.496112 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:43.495459    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:43.496112 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:43.495489    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:43.496112 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:43.495548    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:45.495529122 +0000 UTC m=+2325.758994067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:45.542145 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:45.542113    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:45.542145 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:45.542137    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:45.542574 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:45.542204    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:49.542189756 +0000 UTC m=+2329.805654699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:46.371315 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:46.371282    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:46.371992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:46.371964    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:11:46.467323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:46.467294    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:47.757111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:47.757004    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:47.783125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:47.781820    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:48.546168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.519430    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"]
Oct 30 19:11:48.546168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.520917    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8c7e09ea-3065-4d25-be85-8960f1075956" containerName="revoke-keys"
Oct 30 19:11:48.546168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.520940    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7e09ea-3065-4d25-be85-8960f1075956" containerName="revoke-keys"
Oct 30 19:11:48.546168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.521301    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7e09ea-3065-4d25-be85-8960f1075956" containerName="revoke-keys"
Oct 30 19:11:48.628700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.628659    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"]
Oct 30 19:11:48.628877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.628765    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.634478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.634403    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:11:48.634478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.634473    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:48.668862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.668833    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.669121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.669101    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.669217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.669161    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llj4c\" (UniqueName: \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.770132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.770101    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.770557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.770366    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.770557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.770426    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llj4c\" (UniqueName: \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.770899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.770873    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.772898 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.772875    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.785241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.785216    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llj4c\" (UniqueName: \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\") pod \"maas-api-revoke-sub-36226956-kk7tq\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:48.951576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:48.951501    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:49.324848 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:11:49.324820    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a951824_d0d9_49f7_a574_ee08f231ed28.slice/crio-038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237 WatchSource:0}: Error finding container 038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237: Status 404 returned error can't find the container with id 038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237
Oct 30 19:11:49.325495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:49.325468    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"]
Oct 30 19:11:49.546990 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:49.546965    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:49.546990 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:49.546986    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:49.549712 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:49.547476    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:57.547221441 +0000 UTC m=+2337.810686366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:50.197561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:50.197523    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq" event={"ID":"6a951824-d0d9-49f7-a574-ee08f231ed28","Type":"ContainerStarted","Data":"0f7b9bfc37df33c69cb7236ea8df4a9d8197f597acddbf1024f37fdda34cbff7"}
Oct 30 19:11:50.197974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:50.197587    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq" event={"ID":"6a951824-d0d9-49f7-a574-ee08f231ed28","Type":"ContainerStarted","Data":"038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237"}
Oct 30 19:11:50.220812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:50.220754    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq" podStartSLOduration=2.220739478 podStartE2EDuration="2.220739478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:50.219520111 +0000 UTC m=+2330.482985060" watchObservedRunningTime="2026-10-30 19:11:50.220739478 +0000 UTC m=+2330.484204426"
Oct 30 19:11:51.213272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:51.213236    5065 generic.go:356] "Generic (PLEG): container finished" podID="6a951824-d0d9-49f7-a574-ee08f231ed28" containerID="0f7b9bfc37df33c69cb7236ea8df4a9d8197f597acddbf1024f37fdda34cbff7" exitCode=0
Oct 30 19:11:51.213670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:51.213326    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq" event={"ID":"6a951824-d0d9-49f7-a574-ee08f231ed28","Type":"ContainerDied","Data":"0f7b9bfc37df33c69cb7236ea8df4a9d8197f597acddbf1024f37fdda34cbff7"}
Oct 30 19:11:52.021971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.021928    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" podUID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" containerName="payload-processing" containerID="cri-o://ace94dc4a3226a299e59d4e27eb290b9969b9e75efe34c4ed44a00d0d6366e09" gracePeriod=25
Oct 30 19:11:52.239120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.237536    5065 generic.go:356] "Generic (PLEG): container finished" podID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" containerID="ace94dc4a3226a299e59d4e27eb290b9969b9e75efe34c4ed44a00d0d6366e09" exitCode=0
Oct 30 19:11:52.239120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.237985    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" event={"ID":"bdd5bc69-6cf7-47e9-8338-5fd12da9da13","Type":"ContainerDied","Data":"ace94dc4a3226a299e59d4e27eb290b9969b9e75efe34c4ed44a00d0d6366e09"}
Oct 30 19:11:52.467340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.467309    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:52.555433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.555401    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\") pod \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") "
Oct 30 19:11:52.556001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.555476    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\") pod \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") "
Oct 30 19:11:52.556001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.555549    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\") pod \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") "
Oct 30 19:11:52.556001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.555787    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\") pod \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\" (UID: \"bdd5bc69-6cf7-47e9-8338-5fd12da9da13\") "
Oct 30 19:11:52.556001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.555893    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume" pod "bdd5bc69-6cf7-47e9-8338-5fd12da9da13" (UID: "bdd5bc69-6cf7-47e9-8338-5fd12da9da13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:52.556420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.556233    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume" pod "bdd5bc69-6cf7-47e9-8338-5fd12da9da13" (UID: "bdd5bc69-6cf7-47e9-8338-5fd12da9da13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:52.556968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.556941    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.557140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.556974    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.557958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.557930    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m" pod "bdd5bc69-6cf7-47e9-8338-5fd12da9da13" (UID: "bdd5bc69-6cf7-47e9-8338-5fd12da9da13"). InnerVolumeSpecName "kube-api-access-krt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:52.558034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.557972    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials" pod "bdd5bc69-6cf7-47e9-8338-5fd12da9da13" (UID: "bdd5bc69-6cf7-47e9-8338-5fd12da9da13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:52.582308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.582282    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:52.658577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.658541    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\" (UniqueName: \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\") pod \"6a951824-d0d9-49f7-a574-ee08f231ed28\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") "
Oct 30 19:11:52.658785 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.658639    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\") pod \"6a951824-d0d9-49f7-a574-ee08f231ed28\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") "
Oct 30 19:11:52.658958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.658937    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\") pod \"6a951824-d0d9-49f7-a574-ee08f231ed28\" (UID: \"6a951824-d0d9-49f7-a574-ee08f231ed28\") "
Oct 30 19:11:52.659026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.658966    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca" pod "6a951824-d0d9-49f7-a574-ee08f231ed28" (UID: "6a951824-d0d9-49f7-a574-ee08f231ed28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:52.659804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.659776    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.659933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.659806    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.659933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.659821    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krt6m\" (UniqueName: \"kubernetes.io/projected/bdd5bc69-6cf7-47e9-8338-5fd12da9da13-kube-api-access-krt6m\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.661154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.661053    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca" pod "6a951824-d0d9-49f7-a574-ee08f231ed28" (UID: "6a951824-d0d9-49f7-a574-ee08f231ed28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:52.661254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.661175    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c" pod "6a951824-d0d9-49f7-a574-ee08f231ed28" (UID: "6a951824-d0d9-49f7-a574-ee08f231ed28"). InnerVolumeSpecName "kube-api-access-llj4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:52.757838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.757801    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" podUID="1bff3ee0-8330-4f87-afa4-36c1412583a2" containerName="payload-processing" containerID="cri-o://6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d" gracePeriod=25
Oct 30 19:11:52.761359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.761337    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llj4c\" (UniqueName: \"kubernetes.io/projected/6a951824-d0d9-49f7-a574-ee08f231ed28-kube-api-access-llj4c\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:52.761359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:52.761360    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a951824-d0d9-49f7-a574-ee08f231ed28-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:53.063535 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:53.063503    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:11:53.085331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.085304    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:53.170296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.168533    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\") pod \"1bff3ee0-8330-4f87-afa4-36c1412583a2\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") "
Oct 30 19:11:53.170296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.168665    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\") pod \"1bff3ee0-8330-4f87-afa4-36c1412583a2\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") "
Oct 30 19:11:53.170296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.168961    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\") pod \"1bff3ee0-8330-4f87-afa4-36c1412583a2\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") "
Oct 30 19:11:53.170296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.169037    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\") pod \"1bff3ee0-8330-4f87-afa4-36c1412583a2\" (UID: \"1bff3ee0-8330-4f87-afa4-36c1412583a2\") "
Oct 30 19:11:53.188940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.172691    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj" pod "1bff3ee0-8330-4f87-afa4-36c1412583a2" (UID: "1bff3ee0-8330-4f87-afa4-36c1412583a2"). InnerVolumeSpecName "kube-api-access-62ttj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:53.188940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.184869    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials" pod "1bff3ee0-8330-4f87-afa4-36c1412583a2" (UID: "1bff3ee0-8330-4f87-afa4-36c1412583a2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:53.188940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.185254    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume" pod "1bff3ee0-8330-4f87-afa4-36c1412583a2" (UID: "1bff3ee0-8330-4f87-afa4-36c1412583a2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:53.188940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.185762    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume" pod "1bff3ee0-8330-4f87-afa4-36c1412583a2" (UID: "1bff3ee0-8330-4f87-afa4-36c1412583a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.317473    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.317508    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bff3ee0-8330-4f87-afa4-36c1412583a2-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.317524    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ttj\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-kube-api-access-62ttj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.317542    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bff3ee0-8330-4f87-afa4-36c1412583a2-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.319416    5065 generic.go:356] "Generic (PLEG): container finished" podID="1bff3ee0-8330-4f87-afa4-36c1412583a2" containerID="6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d" exitCode=0
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.319668    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" event={"ID":"1bff3ee0-8330-4f87-afa4-36c1412583a2","Type":"ContainerDied","Data":"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"}
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.319724    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx" event={"ID":"1bff3ee0-8330-4f87-afa4-36c1412583a2","Type":"ContainerDied","Data":"0fe5b34149e152c08992074cb0d8f58342d1aa2084139ad86571d292b5f3d673"}
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.319928    5065 scope.go:122] "RemoveContainer" containerID="6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"
Oct 30 19:11:53.327112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.320155    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"
Oct 30 19:11:53.335117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.328305    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq" event={"ID":"6a951824-d0d9-49f7-a574-ee08f231ed28","Type":"ContainerDied","Data":"038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237"}
Oct 30 19:11:53.335117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.328356    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="038b075cdf4777099499fbbb7ae6e4b5a03677dc7d7e2505f16be6514cc90237"
Oct 30 19:11:53.335117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.328416    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"
Oct 30 19:11:53.349017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.348787    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t" event={"ID":"bdd5bc69-6cf7-47e9-8338-5fd12da9da13","Type":"ContainerDied","Data":"b100fe77e0d73cc98941cf7a96feacfc62f86038a5d92fbb8289e3cd93c9dbea"}
Oct 30 19:11:53.349017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.348879    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"
Oct 30 19:11:53.366622 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.366603    5065 scope.go:122] "RemoveContainer" containerID="6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"
Oct 30 19:11:53.366931 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:53.366911    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d\": container with ID starting with 6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d not found: ID does not exist" containerID="6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"
Oct 30 19:11:53.367046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.366942    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d"} err="failed to get container status \"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d\": rpc error: code = NotFound desc = could not find container \"6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d\": container with ID starting with 6bc49295318e01ceaaba62e472d5b757566c3a892e4997d7f3ffc8051fc8d56d not found: ID does not exist"
Oct 30 19:11:53.367046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.366965    5065 scope.go:122] "RemoveContainer" containerID="ace94dc4a3226a299e59d4e27eb290b9969b9e75efe34c4ed44a00d0d6366e09"
Oct 30 19:11:53.390742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.390714    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:53.397261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.397237    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dxpb7t"]
Oct 30 19:11:53.411969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.411948    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:53.417120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.417073    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57d8kkwkx"]
Oct 30 19:11:53.896639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.896603    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bff3ee0-8330-4f87-afa4-36c1412583a2" path="/var/lib/kubelet/pods/1bff3ee0-8330-4f87-afa4-36c1412583a2/volumes"
Oct 30 19:11:53.897263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.897244    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" path="/var/lib/kubelet/pods/bdd5bc69-6cf7-47e9-8338-5fd12da9da13/volumes"
Oct 30 19:11:53.949729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:53.949706    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:11:55.411599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:11:55.407670    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:11:57.617207 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:57.617178    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:57.617207 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:57.617201    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:11:57.617662 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:11:57.617253    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials podName:61b205bc-a004-4928-8d8c-47888b37d898 nodeName:}" failed. No retries permitted until 2026-10-30 19:12:13.617239726 +0000 UTC m=+2353.880704653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" (UID: "61b205bc-a004-4928-8d8c-47888b37d898") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:12:00.408471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.408427    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" podUID="61b205bc-a004-4928-8d8c-47888b37d898" containerName="payload-processing" containerID="cri-o://d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a" gracePeriod=25
Oct 30 19:12:00.796518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.796487    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.938281    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\") pod \"61b205bc-a004-4928-8d8c-47888b37d898\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") "
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.938455    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\") pod \"61b205bc-a004-4928-8d8c-47888b37d898\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") "
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.938585    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\") pod \"61b205bc-a004-4928-8d8c-47888b37d898\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") "
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.938816    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\") pod \"61b205bc-a004-4928-8d8c-47888b37d898\" (UID: \"61b205bc-a004-4928-8d8c-47888b37d898\") "
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.939015    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume" pod "61b205bc-a004-4928-8d8c-47888b37d898" (UID: "61b205bc-a004-4928-8d8c-47888b37d898"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.940180    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume" pod "61b205bc-a004-4928-8d8c-47888b37d898" (UID: "61b205bc-a004-4928-8d8c-47888b37d898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:00.940999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.940270    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:00.944572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.944542    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials" pod "61b205bc-a004-4928-8d8c-47888b37d898" (UID: "61b205bc-a004-4928-8d8c-47888b37d898"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:00.944703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:00.944590    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr" pod "61b205bc-a004-4928-8d8c-47888b37d898" (UID: "61b205bc-a004-4928-8d8c-47888b37d898"). InnerVolumeSpecName "kube-api-access-wm7dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:01.042386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.042360    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b205bc-a004-4928-8d8c-47888b37d898-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:01.042386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.042385    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:01.042589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.042396    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7dr\" (UniqueName: \"kubernetes.io/projected/61b205bc-a004-4928-8d8c-47888b37d898-kube-api-access-wm7dr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:01.463497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.463374    5065 generic.go:356] "Generic (PLEG): container finished" podID="61b205bc-a004-4928-8d8c-47888b37d898" containerID="d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a" exitCode=0
Oct 30 19:12:01.464908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.463699    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" event={"ID":"61b205bc-a004-4928-8d8c-47888b37d898","Type":"ContainerDied","Data":"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"}
Oct 30 19:12:01.464908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.463779    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj" event={"ID":"61b205bc-a004-4928-8d8c-47888b37d898","Type":"ContainerDied","Data":"ad97abfa4b7bb3ca1dedaa5a5d40d249e032affeafa39ddf6d0d6a5bf669017c"}
Oct 30 19:12:01.464908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.463818    5065 scope.go:122] "RemoveContainer" containerID="d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"
Oct 30 19:12:01.464908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.464033    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"
Oct 30 19:12:01.573261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.573238    5065 scope.go:122] "RemoveContainer" containerID="d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"
Oct 30 19:12:01.573854 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:12:01.573820    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a\": container with ID starting with d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a not found: ID does not exist" containerID="d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"
Oct 30 19:12:01.573960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.573865    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a"} err="failed to get container status \"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a\": rpc error: code = NotFound desc = could not find container \"d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a\": container with ID starting with d641f56e712a6cb382d311f12ce9c78c75a30e4ce0ff742138974692e71cff5a not found: ID does not exist"
Oct 30 19:12:01.587586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.587544    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:12:01.596295 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.596265    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78b5kwj"]
Oct 30 19:12:01.893227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:01.893197    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b205bc-a004-4928-8d8c-47888b37d898" path="/var/lib/kubelet/pods/61b205bc-a004-4928-8d8c-47888b37d898/volumes"
Oct 30 19:12:12.894576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:12.894541    5065 scope.go:122] "RemoveContainer" containerID="1ba8bdfb3a54f3ade255a24460f65af56b586398d488efc178e77c146e666028"
Oct 30 19:12:15.735115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:15.719564    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:15.814109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:15.813177    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.113280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.113236    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.161109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.114568    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.297527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.297483    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.355113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.345028    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:12:16.355113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.345341    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy" containerID="cri-o://ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c" gracePeriod=30
Oct 30 19:12:16.820112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.816191    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.820112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:16.816807    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:12:17.019452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.019362    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:17.052434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.038486    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:12:17.148247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.147492    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:17.298133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.254373    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:17.300876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.299111    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:17.329268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.329239    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:12:17.329579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.329564    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:12:17.329733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.329722    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:12:17.555471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.555440    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:12:17.575639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.575610    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:12:17.650935 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.650904    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:12:17.679570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:17.679502    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:12:18.195136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.183377    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:12:18.218128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.206025    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1bff3ee0-8330-4f87-afa4-36c1412583a2" containerName="payload-processing"
Oct 30 19:12:18.223273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.223245    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bff3ee0-8330-4f87-afa4-36c1412583a2" containerName="payload-processing"
Oct 30 19:12:18.223538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.223520    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" containerName="payload-processing"
Oct 30 19:12:18.223662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.223648    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" containerName="payload-processing"
Oct 30 19:12:18.223846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.223832    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6a951824-d0d9-49f7-a574-ee08f231ed28" containerName="revoke-keys"
Oct 30 19:12:18.223955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.223942    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a951824-d0d9-49f7-a574-ee08f231ed28" containerName="revoke-keys"
Oct 30 19:12:18.238614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.238585    5065 cpu_manager.go:414] "containerMap: removing container" podUID="61b205bc-a004-4928-8d8c-47888b37d898" containerName="payload-processing"
Oct 30 19:12:18.238839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.238821    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b205bc-a004-4928-8d8c-47888b37d898" containerName="payload-processing"
Oct 30 19:12:18.239371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.239344    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a951824-d0d9-49f7-a574-ee08f231ed28" containerName="revoke-keys"
Oct 30 19:12:18.285527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.285501    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b205bc-a004-4928-8d8c-47888b37d898" containerName="payload-processing"
Oct 30 19:12:18.285800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.285777    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bff3ee0-8330-4f87-afa4-36c1412583a2" containerName="payload-processing"
Oct 30 19:12:18.285954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.285940    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd5bc69-6cf7-47e9-8338-5fd12da9da13" containerName="payload-processing"
Oct 30 19:12:18.307477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.307445    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:12:18.307887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.307867    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.387428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.387396    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:12:18.414120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.387778    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:12:18.416152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.387781    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-e55b23f0af98\""
Oct 30 19:12:18.416628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.416609    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:12:18.586539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.440516    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-cpnpr\""
Oct 30 19:12:18.586539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.440580    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:12:18.586539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.473869    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.586539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.584519    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.586539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.584825    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.627145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.619421    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxt2\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.736134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.735270    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxt2\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.736134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.735547    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.736134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.735673    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.736134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.735786    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.740556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.736659    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.740556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.737656    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.764155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.758994    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.785042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.785018    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxt2\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:18.921327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:18.921297    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:19.256389 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:12:19.256346    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53022857_f972_44e2_8aa4_4aae0700fc8e.slice/crio-545be99c36a1597567bcc8c4357439a45de9b0ccdf17b156eceab1eb609dd916 WatchSource:0}: Error finding container 545be99c36a1597567bcc8c4357439a45de9b0ccdf17b156eceab1eb609dd916: Status 404 returned error can't find the container with id 545be99c36a1597567bcc8c4357439a45de9b0ccdf17b156eceab1eb609dd916
Oct 30 19:12:19.261388 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:19.261199    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:12:19.894919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:19.894881    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" event={"ID":"53022857-f972-44e2-8aa4-4aae0700fc8e","Type":"ContainerStarted","Data":"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"}
Oct 30 19:12:19.895129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:19.894950    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:19.895129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:19.894971    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" event={"ID":"53022857-f972-44e2-8aa4-4aae0700fc8e","Type":"ContainerStarted","Data":"545be99c36a1597567bcc8c4357439a45de9b0ccdf17b156eceab1eb609dd916"}
Oct 30 19:12:19.938820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:19.938751    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" podStartSLOduration=1.938732881 podStartE2EDuration="1.938732881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:19.928385622 +0000 UTC m=+2360.191850598" watchObservedRunningTime="2026-10-30 19:12:19.938732881 +0000 UTC m=+2360.202197836"
Oct 30 19:12:20.859144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.859110    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:12:20.864814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.864795    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:20.872015 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.871523    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"agc-cred-eec5675f2a95\""
Oct 30 19:12:20.878972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.878942    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:12:20.920356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.920318    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:20.920556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.920383    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m272q\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:20.920759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.920732    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:20.920868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:20.920784    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.022125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.022074    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.022303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.022161    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m272q\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.022933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.022903    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.023070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.022934    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.023070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.022983    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.023665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.023639    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.026003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.025977    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.036174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.036148    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m272q\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.185626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.185541    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.540342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.540259    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:12:21.540440 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:12:21.540372    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5949fb9f_1086_429c_bb9d_13c952bd7b9b.slice/crio-4eadecac9947bebb169031c92c35a7e30daafb160f4855a15966dedd74bc7c29 WatchSource:0}: Error finding container 4eadecac9947bebb169031c92c35a7e30daafb160f4855a15966dedd74bc7c29: Status 404 returned error can't find the container with id 4eadecac9947bebb169031c92c35a7e30daafb160f4855a15966dedd74bc7c29
Oct 30 19:12:21.923573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.923481    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" event={"ID":"5949fb9f-1086-429c-bb9d-13c952bd7b9b","Type":"ContainerStarted","Data":"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"}
Oct 30 19:12:21.923573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.923545    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" event={"ID":"5949fb9f-1086-429c-bb9d-13c952bd7b9b","Type":"ContainerStarted","Data":"4eadecac9947bebb169031c92c35a7e30daafb160f4855a15966dedd74bc7c29"}
Oct 30 19:12:21.924114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.923598    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:21.946569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:21.946524    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" podStartSLOduration=1.946508444 podStartE2EDuration="1.946508444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:21.945600512 +0000 UTC m=+2362.209065459" watchObservedRunningTime="2026-10-30 19:12:21.946508444 +0000 UTC m=+2362.209973401"
Oct 30 19:12:22.641530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.641505    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:12:22.731284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731257    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731340    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731373    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731431    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731496    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731520    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731560    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731644    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.731788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.731757    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\") pod \"00becf78-a3bf-4042-b3dc-8c396c026f44\" (UID: \"00becf78-a3bf-4042-b3dc-8c396c026f44\") "
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.732612    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.732805    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00becf78-a3bf-4042-b3dc-8c396c026f44-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.732914    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.733027    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.733813    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.734013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.733976    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.735632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.734621    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:12:22.735632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.734966    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.735632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.735011    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:22.736187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.736153    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb" pod "00becf78-a3bf-4042-b3dc-8c396c026f44" (UID: "00becf78-a3bf-4042-b3dc-8c396c026f44"). InnerVolumeSpecName "kube-api-access-bfkqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:22.834825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834791    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.834825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834826    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834841    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834855    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834871    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834884    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834897    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.835047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.834910    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkqb\" (UniqueName: \"kubernetes.io/projected/00becf78-a3bf-4042-b3dc-8c396c026f44-kube-api-access-bfkqb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.936409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.936316    5065 generic.go:356] "Generic (PLEG): container finished" podID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerID="ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c" exitCode=0
Oct 30 19:12:22.936409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.936375    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"
Oct 30 19:12:22.936409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.936396    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerDied","Data":"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"}
Oct 30 19:12:22.936992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.936444    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq" event={"ID":"00becf78-a3bf-4042-b3dc-8c396c026f44","Type":"ContainerDied","Data":"86b222059b26192daef3181571b816347da745f3dbbc64aa042366a9ee5c36f6"}
Oct 30 19:12:22.936992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.936466    5065 scope.go:122] "RemoveContainer" containerID="ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"
Oct 30 19:12:22.950715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.950696    5065 scope.go:122] "RemoveContainer" containerID="fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"
Oct 30 19:12:22.963274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.963235    5065 scope.go:122] "RemoveContainer" containerID="ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"
Oct 30 19:12:22.963516 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:12:22.963498    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c\": container with ID starting with ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c not found: ID does not exist" containerID="ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"
Oct 30 19:12:22.963581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.963521    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c"} err="failed to get container status \"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c\": rpc error: code = NotFound desc = could not find container \"ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c\": container with ID starting with ce4e254607b6a2aaef6fdc1fedfd74b8ed7d13173067b5d8e917e669b7ea9e9c not found: ID does not exist"
Oct 30 19:12:22.963581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.963538    5065 scope.go:122] "RemoveContainer" containerID="fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"
Oct 30 19:12:22.963829 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:12:22.963805    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2\": container with ID starting with fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2 not found: ID does not exist" containerID="fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"
Oct 30 19:12:22.963873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.963838    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2"} err="failed to get container status \"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2\": rpc error: code = NotFound desc = could not find container \"fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2\": container with ID starting with fa4decea8065b9e8f1c121dd32b5210ab22ede04a646f32ca7c12c21fedb33b2 not found: ID does not exist"
Oct 30 19:12:22.967335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.967315    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:12:22.976486 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:22.976464    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-cv6jq"]
Oct 30 19:12:23.059789 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:12:23.059748    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:12:23.894980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:23.894947    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" path="/var/lib/kubelet/pods/00becf78-a3bf-4042-b3dc-8c396c026f44/volumes"
Oct 30 19:12:30.917888 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:30.917856    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:12:32.939598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:32.939570    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:12:33.033061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:33.033029    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:12:38.034024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.033983    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" podUID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" containerName="payload-processing" containerID="cri-o://6e015a496198acbbb349017d8eb88a7ebc961b74b8175dba3b8713b44463e920" gracePeriod=25
Oct 30 19:12:38.168226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.168195    5065 generic.go:356] "Generic (PLEG): container finished" podID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" containerID="6e015a496198acbbb349017d8eb88a7ebc961b74b8175dba3b8713b44463e920" exitCode=0
Oct 30 19:12:38.168317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.168267    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" event={"ID":"ce87ce53-3994-4f05-a85e-ca01b0a7df31","Type":"ContainerDied","Data":"6e015a496198acbbb349017d8eb88a7ebc961b74b8175dba3b8713b44463e920"}
Oct 30 19:12:38.371074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.371043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:12:38.463060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463025    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\") pod \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") "
Oct 30 19:12:38.463275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463163    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\") pod \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") "
Oct 30 19:12:38.463275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463195    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\") pod \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") "
Oct 30 19:12:38.463427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463408    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\") pod \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\" (UID: \"ce87ce53-3994-4f05-a85e-ca01b0a7df31\") "
Oct 30 19:12:38.463648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463617    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume" pod "ce87ce53-3994-4f05-a85e-ca01b0a7df31" (UID: "ce87ce53-3994-4f05-a85e-ca01b0a7df31"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:38.463770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.463693    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume" pod "ce87ce53-3994-4f05-a85e-ca01b0a7df31" (UID: "ce87ce53-3994-4f05-a85e-ca01b0a7df31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:38.464517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.464488    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:38.464640 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.464521    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce87ce53-3994-4f05-a85e-ca01b0a7df31-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:38.465918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.465890    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials" pod "ce87ce53-3994-4f05-a85e-ca01b0a7df31" (UID: "ce87ce53-3994-4f05-a85e-ca01b0a7df31"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:38.466021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.465945    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv" pod "ce87ce53-3994-4f05-a85e-ca01b0a7df31" (UID: "ce87ce53-3994-4f05-a85e-ca01b0a7df31"). InnerVolumeSpecName "kube-api-access-9bflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:38.567347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.567312    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bflv\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-kube-api-access-9bflv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:38.567347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:38.567349    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce87ce53-3994-4f05-a85e-ca01b0a7df31-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:39.179306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.179268    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7" event={"ID":"ce87ce53-3994-4f05-a85e-ca01b0a7df31","Type":"ContainerDied","Data":"024f6d1d28eb3581cc89eaed2e9b8ad0705427c58e0700392392acc6415739a4"}
Oct 30 19:12:39.179759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.179346    5065 scope.go:122] "RemoveContainer" containerID="6e015a496198acbbb349017d8eb88a7ebc961b74b8175dba3b8713b44463e920"
Oct 30 19:12:39.179759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.179280    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"
Oct 30 19:12:39.220556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.220499    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:12:39.229887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.229852    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45h8xt7"]
Oct 30 19:12:39.895179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:39.895149    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" path="/var/lib/kubelet/pods/ce87ce53-3994-4f05-a85e-ca01b0a7df31/volumes"
Oct 30 19:12:51.127593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.127320    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"]
Oct 30 19:12:51.129674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.129634    5065 cpu_manager.go:414] "containerMap: removing container" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.129873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.129856    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.129989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.129978    5065 cpu_manager.go:414] "containerMap: removing container" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.130068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130058    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.130288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130272    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" containerName="payload-processing"
Oct 30 19:12:51.130407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130395    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" containerName="payload-processing"
Oct 30 19:12:51.130710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130693    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce87ce53-3994-4f05-a85e-ca01b0a7df31" containerName="payload-processing"
Oct 30 19:12:51.130827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130816    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.130916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.130907    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="00becf78-a3bf-4042-b3dc-8c396c026f44" containerName="istio-proxy"
Oct 30 19:12:51.154826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.154794    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"]
Oct 30 19:12:51.155223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.155204    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.163583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.163559    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:51.163943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.163915    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:12:51.181921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.181883    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.182201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.182181    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.182362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.182274    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hblh\" (UniqueName: \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.284209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.284173    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.284398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.284282    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hblh\" (UniqueName: \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.284475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.284428    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.284886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.284864    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.287011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.286987    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.298913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.298889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hblh\" (UniqueName: \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\") pod \"maas-api-revoke-sub-58a8e268-t8bb8\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.475593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.475519    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:51.772222 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:12:51.772194    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfa8120_bf52_46b3_b1aa_f120a0513842.slice/crio-6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c WatchSource:0}: Error finding container 6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c: Status 404 returned error can't find the container with id 6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c
Oct 30 19:12:51.774378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:51.774351    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"]
Oct 30 19:12:52.494403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:52.494279    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8" event={"ID":"9bfa8120-bf52-46b3-b1aa-f120a0513842","Type":"ContainerStarted","Data":"05fd1bfcbd7d516a9701acae86fbebe7dc9b6c32a647522e899b15f579357e59"}
Oct 30 19:12:52.494403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:52.494345    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8" event={"ID":"9bfa8120-bf52-46b3-b1aa-f120a0513842","Type":"ContainerStarted","Data":"6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c"}
Oct 30 19:12:52.520165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:52.520115    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8" podStartSLOduration=1.520102621 podStartE2EDuration="1.520102621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:52.516698257 +0000 UTC m=+2392.780163206" watchObservedRunningTime="2026-10-30 19:12:52.520102621 +0000 UTC m=+2392.783567562"
Oct 30 19:12:53.100486 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:12:53.100360    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:12:53.134168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.134131    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"]
Oct 30 19:12:53.159406 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.159369    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.171641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.171619    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"]
Oct 30 19:12:53.217179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.217145    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwt55\" (UniqueName: \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.217363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.217294    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.217703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.217681    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.319358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.319330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.319521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.319403    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwt55\" (UniqueName: \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.319594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.319523    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.320397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.320367    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.322029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.322005    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.333232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.333210    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwt55\" (UniqueName: \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\") pod \"maas-api-revoke-sub-e4aee02a-fztm5\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.480674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.480595    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:53.526433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.526401    5065 generic.go:356] "Generic (PLEG): container finished" podID="9bfa8120-bf52-46b3-b1aa-f120a0513842" containerID="05fd1bfcbd7d516a9701acae86fbebe7dc9b6c32a647522e899b15f579357e59" exitCode=0
Oct 30 19:12:53.526993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.526476    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8" event={"ID":"9bfa8120-bf52-46b3-b1aa-f120a0513842","Type":"ContainerDied","Data":"05fd1bfcbd7d516a9701acae86fbebe7dc9b6c32a647522e899b15f579357e59"}
Oct 30 19:12:53.790521 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:12:53.790487    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f25bcbf_61ed_48d7_a70c_a3fdd2938c10.slice/crio-61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead WatchSource:0}: Error finding container 61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead: Status 404 returned error can't find the container with id 61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead
Oct 30 19:12:53.791496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:53.791374    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"]
Oct 30 19:12:54.537657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.537622    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5" event={"ID":"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10","Type":"ContainerStarted","Data":"e42a5a8de1c2f54619ef5c9595303e1bc222680feed4ff0745b27606d0fbb247"}
Oct 30 19:12:54.538215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.537689    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5" event={"ID":"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10","Type":"ContainerStarted","Data":"61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead"}
Oct 30 19:12:54.561786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.561726    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5" podStartSLOduration=1.561712105 podStartE2EDuration="1.561712105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:54.559577453 +0000 UTC m=+2394.823042402" watchObservedRunningTime="2026-10-30 19:12:54.561712105 +0000 UTC m=+2394.825177054"
Oct 30 19:12:54.728709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.728685    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:54.846841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.846810    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\") pod \"9bfa8120-bf52-46b3-b1aa-f120a0513842\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") "
Oct 30 19:12:54.846997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.846896    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\" (UniqueName: \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\") pod \"9bfa8120-bf52-46b3-b1aa-f120a0513842\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") "
Oct 30 19:12:54.847075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.847064    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\") pod \"9bfa8120-bf52-46b3-b1aa-f120a0513842\" (UID: \"9bfa8120-bf52-46b3-b1aa-f120a0513842\") "
Oct 30 19:12:54.847401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.847373    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca" pod "9bfa8120-bf52-46b3-b1aa-f120a0513842" (UID: "9bfa8120-bf52-46b3-b1aa-f120a0513842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:54.848290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.848259    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:54.849363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.849342    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca" pod "9bfa8120-bf52-46b3-b1aa-f120a0513842" (UID: "9bfa8120-bf52-46b3-b1aa-f120a0513842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:54.849468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.849448    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh" pod "9bfa8120-bf52-46b3-b1aa-f120a0513842" (UID: "9bfa8120-bf52-46b3-b1aa-f120a0513842"). InnerVolumeSpecName "kube-api-access-2hblh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:54.962965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.953445    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hblh\" (UniqueName: \"kubernetes.io/projected/9bfa8120-bf52-46b3-b1aa-f120a0513842-kube-api-access-2hblh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:54.963293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:54.963264    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bfa8120-bf52-46b3-b1aa-f120a0513842-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:55.547915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:55.547881    5065 generic.go:356] "Generic (PLEG): container finished" podID="7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" containerID="e42a5a8de1c2f54619ef5c9595303e1bc222680feed4ff0745b27606d0fbb247" exitCode=0
Oct 30 19:12:55.548412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:55.547978    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5" event={"ID":"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10","Type":"ContainerDied","Data":"e42a5a8de1c2f54619ef5c9595303e1bc222680feed4ff0745b27606d0fbb247"}
Oct 30 19:12:55.549530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:55.549504    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"
Oct 30 19:12:55.549665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:55.549545    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8" event={"ID":"9bfa8120-bf52-46b3-b1aa-f120a0513842","Type":"ContainerDied","Data":"6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c"}
Oct 30 19:12:55.549665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:55.549582    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6528aa6061bc34388207a5f273b7aed81c2c6ac42d6c170fc9b91d2e4552610c"
Oct 30 19:12:56.860767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:56.860742    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:57.007074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.006981    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\") pod \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") "
Oct 30 19:12:57.007270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.007097    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\" (UniqueName: \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\") pod \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") "
Oct 30 19:12:57.007270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.007251    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\") pod \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\" (UID: \"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10\") "
Oct 30 19:12:57.007411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.007382    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca" pod "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" (UID: "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:57.008202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.008175    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:57.010170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.010145    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca" pod "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" (UID: "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:57.010289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.010199    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55" pod "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" (UID: "7f25bcbf-61ed-48d7-a70c-a3fdd2938c10"). InnerVolumeSpecName "kube-api-access-mwt55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:57.109406 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.109377    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwt55\" (UniqueName: \"kubernetes.io/projected/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-kube-api-access-mwt55\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:57.109406 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.109405    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:57.637537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.637507    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"
Oct 30 19:12:57.637725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.637555    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5" event={"ID":"7f25bcbf-61ed-48d7-a70c-a3fdd2938c10","Type":"ContainerDied","Data":"61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead"}
Oct 30 19:12:57.637725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:12:57.637604    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b6fe61ec30d564d3249d4e064ea349a6e4e992738449589ed268e967e83ead"
Oct 30 19:13:01.684820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.684782    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"]
Oct 30 19:13:01.685781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685762    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9bfa8120-bf52-46b3-b1aa-f120a0513842" containerName="revoke-keys"
Oct 30 19:13:01.685781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685782    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfa8120-bf52-46b3-b1aa-f120a0513842" containerName="revoke-keys"
Oct 30 19:13:01.685925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685802    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" containerName="revoke-keys"
Oct 30 19:13:01.685925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685808    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" containerName="revoke-keys"
Oct 30 19:13:01.686045 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685950    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfa8120-bf52-46b3-b1aa-f120a0513842" containerName="revoke-keys"
Oct 30 19:13:01.686045 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.685962    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" containerName="revoke-keys"
Oct 30 19:13:01.690456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.690434    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.696918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.696890    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:13:01.697057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.697013    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:01.704843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.704816    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"]
Oct 30 19:13:01.820693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.807235    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.821650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.821598    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.822231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.822187    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss955\" (UniqueName: \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.923844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.923807    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.924280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.924257    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss955\" (UniqueName: \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.924637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.924613    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.925377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.925353    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.926588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.926568    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:01.938444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:01.938381    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss955\" (UniqueName: \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\") pod \"maas-api-revoke-sub-57d134ea-qcf2z\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:02.009050 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:02.009016    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:02.302633 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:02.302600    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7633aea0_4d0f_494f_ade4_ffdcc05528db.slice/crio-75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319 WatchSource:0}: Error finding container 75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319: Status 404 returned error can't find the container with id 75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319
Oct 30 19:13:02.303825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:02.303801    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"]
Oct 30 19:13:02.727875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:02.727843    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z" event={"ID":"7633aea0-4d0f-494f-ade4-ffdcc05528db","Type":"ContainerStarted","Data":"75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319"}
Oct 30 19:13:03.759944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.759505    5065 generic.go:356] "Generic (PLEG): container finished" podID="7633aea0-4d0f-494f-ade4-ffdcc05528db" containerID="cc6c322daab5c350ecc202da2e1c798a288fb42b145b536be3c6d5c9e9a1703b" exitCode=0
Oct 30 19:13:03.759944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.759741    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z" event={"ID":"7633aea0-4d0f-494f-ade4-ffdcc05528db","Type":"ContainerDied","Data":"cc6c322daab5c350ecc202da2e1c798a288fb42b145b536be3c6d5c9e9a1703b"}
Oct 30 19:13:03.798161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.798131    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:13:03.825498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.825465    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:13:03.918703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.918676    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:13:03.927898 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:03.927871    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:13:04.373110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.370148    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:04.541111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.540473    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:04.541111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.540612    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.617133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.616101    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.617133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.616406    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hx5\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.617133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.616486    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.617133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.616583    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.790755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.783303    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hx5\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.790755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.783399    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.790755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.783494    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.790755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.783860    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.850110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.836971    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.850110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.837567    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.900138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.892999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hx5\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:04.936121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:04.917469    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:05.099773 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.099724    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:05.149897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.149672    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"]
Oct 30 19:13:05.268198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.262374    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"]
Oct 30 19:13:05.268198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.262496    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.346245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.346196    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9tq\" (UniqueName: \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.346522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.346508    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.346715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.346702    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.448651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.448599    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9tq\" (UniqueName: \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.448855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.448668    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.448855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.448784    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.449321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.449295    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.452715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.452681    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.463830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.463806    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9tq\" (UniqueName: \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\") pod \"maas-api-revoke-sub-5c4c1e80-7sw9h\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.535203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.535175    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:05.550704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.550681    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\") pod \"7633aea0-4d0f-494f-ade4-ffdcc05528db\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") "
Oct 30 19:13:05.550830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.550777    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\" (UniqueName: \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\") pod \"7633aea0-4d0f-494f-ade4-ffdcc05528db\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") "
Oct 30 19:13:05.550973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.550962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\") pod \"7633aea0-4d0f-494f-ade4-ffdcc05528db\" (UID: \"7633aea0-4d0f-494f-ade4-ffdcc05528db\") "
Oct 30 19:13:05.551019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.550992    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca" pod "7633aea0-4d0f-494f-ade4-ffdcc05528db" (UID: "7633aea0-4d0f-494f-ade4-ffdcc05528db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:05.551654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.551633    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:05.553339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.553315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca" pod "7633aea0-4d0f-494f-ade4-ffdcc05528db" (UID: "7633aea0-4d0f-494f-ade4-ffdcc05528db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:05.553438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.553413    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955" pod "7633aea0-4d0f-494f-ade4-ffdcc05528db" (UID: "7633aea0-4d0f-494f-ade4-ffdcc05528db"). InnerVolumeSpecName "kube-api-access-ss955". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:05.592389 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.592361    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:05.612441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.612412    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:05.653199 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.653168    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7633aea0-4d0f-494f-ade4-ffdcc05528db-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:05.653334 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:05.653203    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss955\" (UniqueName: \"kubernetes.io/projected/7633aea0-4d0f-494f-ade4-ffdcc05528db-kube-api-access-ss955\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:06.003607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.003582    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"]
Oct 30 19:13:06.287014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.286983    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"
Oct 30 19:13:06.287190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.286997    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z" event={"ID":"7633aea0-4d0f-494f-ade4-ffdcc05528db","Type":"ContainerDied","Data":"75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319"}
Oct 30 19:13:06.287190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.287105    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ce97eb59029fb7d3a9762be4fdf34f3eabc2ff24b83d38105f8e2caf4df319"
Oct 30 19:13:06.289433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.289297    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" event={"ID":"7783e6d4-db63-480c-9be3-419910989652","Type":"ContainerStarted","Data":"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"}
Oct 30 19:13:06.289433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.289350    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" event={"ID":"7783e6d4-db63-480c-9be3-419910989652","Type":"ContainerStarted","Data":"b87825783e0119952f9e4f51e0ee252af00186d0ef98b9a6cebb3c89feac89f3"}
Oct 30 19:13:06.289757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.289727    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:06.291160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.291133    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h" event={"ID":"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea","Type":"ContainerStarted","Data":"b14fa0e43c1b553c203c4b48dbd7d5b16b487d275a1f43475f42b50b9b639390"}
Oct 30 19:13:06.313394 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:06.313351    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" podStartSLOduration=2.313337862 podStartE2EDuration="2.313337862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:06.310733206 +0000 UTC m=+2406.574198167" watchObservedRunningTime="2026-10-30 19:13:06.313337862 +0000 UTC m=+2406.576802809"
Oct 30 19:13:07.304514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:07.304475    5065 generic.go:356] "Generic (PLEG): container finished" podID="fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" containerID="1e2466150a5ee99920e7df424b7e5bc546235246f29c3fff840bbf3a2558ee16" exitCode=0
Oct 30 19:13:07.304950 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:07.304563    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h" event={"ID":"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea","Type":"ContainerDied","Data":"1e2466150a5ee99920e7df424b7e5bc546235246f29c3fff840bbf3a2558ee16"}
Oct 30 19:13:08.574648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.574592    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:08.679577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.679544    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\" (UniqueName: \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\") pod \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") "
Oct 30 19:13:08.679769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.679618    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\") pod \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") "
Oct 30 19:13:08.679840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.679811    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\") pod \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\" (UID: \"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea\") "
Oct 30 19:13:08.681964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.680561    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca" pod "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" (UID: "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:08.681964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.680744    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:08.683301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.683266    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca" pod "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" (UID: "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:08.683501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.683483    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq" pod "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" (UID: "fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea"). InnerVolumeSpecName "kube-api-access-qz9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:08.782730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.782700    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:08.782730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:08.782728    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9tq\" (UniqueName: \"kubernetes.io/projected/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea-kube-api-access-qz9tq\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:09.329494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:09.329464    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"
Oct 30 19:13:09.329805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:09.329492    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h" event={"ID":"fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea","Type":"ContainerDied","Data":"b14fa0e43c1b553c203c4b48dbd7d5b16b487d275a1f43475f42b50b9b639390"}
Oct 30 19:13:09.329805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:09.329539    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14fa0e43c1b553c203c4b48dbd7d5b16b487d275a1f43475f42b50b9b639390"
Oct 30 19:13:12.937109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.936734    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954211    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7633aea0-4d0f-494f-ade4-ffdcc05528db" containerName="revoke-keys"
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954246    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7633aea0-4d0f-494f-ade4-ffdcc05528db" containerName="revoke-keys"
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954344    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" containerName="revoke-keys"
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954353    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" containerName="revoke-keys"
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954588    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7633aea0-4d0f-494f-ade4-ffdcc05528db" containerName="revoke-keys"
Oct 30 19:13:12.957115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.954614    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" containerName="revoke-keys"
Oct 30 19:13:12.982109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:12.977346    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.036910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.029378    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a717c7\""
Oct 30 19:13:13.044319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.040246    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-e2e--ebe4ecff-dockercfg-vjqtd\""
Oct 30 19:13:13.044319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.040303    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"agc-cred-56b60f463986\""
Oct 30 19:13:13.044319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.040406    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"kube-root-ca.crt\""
Oct 30 19:13:13.044319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.040562    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"routing-overlay\""
Oct 30 19:13:13.044319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.040603    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"openshift-service-ca.crt\""
Oct 30 19:13:13.062234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.062126    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:13.069170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.069137    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.069639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.069610    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.069876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.069857    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.070187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.070165    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmx5\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.087427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.075671    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"]
Oct 30 19:13:13.087427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.082265    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"]
Oct 30 19:13:13.087427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.082367    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.101780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.101747    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:13:13.101901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.101807    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:13.171501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171468    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.171613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171559    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.171659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171616    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.171727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171695    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjrp\" (UniqueName: \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.171939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171920    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.171977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.171954    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmx5\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.172140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.172120    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.172400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.172377    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.172826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.172803    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.174075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.174055    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.191291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.191233    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmx5\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.274013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.273984    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.274179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.274058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjrp\" (UniqueName: \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.274327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.274207    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.274845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.274779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.276439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.276418    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.287942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.287918    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjrp\" (UniqueName: \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\") pod \"maas-api-revoke-sub-65207ead-xvhmn\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.385416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.385387    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:13.427392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.427360    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:13.840280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.839194    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:13.918980 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:13.918936    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f96907_bbba_4cf5_8c22_10c98556a125.slice/crio-aa53917cb04fc7eec0e1e5ce7e21a1a77a9a11afb4096e14c8a2cdf3a810652a WatchSource:0}: Error finding container aa53917cb04fc7eec0e1e5ce7e21a1a77a9a11afb4096e14c8a2cdf3a810652a: Status 404 returned error can't find the container with id aa53917cb04fc7eec0e1e5ce7e21a1a77a9a11afb4096e14c8a2cdf3a810652a
Oct 30 19:13:13.942544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.942516    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:13.944764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:13.944743    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"]
Oct 30 19:13:13.946987 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:13.946966    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc2a93e_9613_4fe8_8aee_d6a2ab6d086d.slice/crio-ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5 WatchSource:0}: Error finding container ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5: Status 404 returned error can't find the container with id ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5
Oct 30 19:13:14.453213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:14.453123    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn" event={"ID":"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d","Type":"ContainerStarted","Data":"ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5"}
Oct 30 19:13:14.455173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:14.455140    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" event={"ID":"d4f96907-bbba-4cf5-8c22-10c98556a125","Type":"ContainerStarted","Data":"28479db9dadae387264bb39236e62fda16f7b24c7c644931d54bf9cc069469a9"}
Oct 30 19:13:14.455319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:14.455197    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" event={"ID":"d4f96907-bbba-4cf5-8c22-10c98556a125","Type":"ContainerStarted","Data":"aa53917cb04fc7eec0e1e5ce7e21a1a77a9a11afb4096e14c8a2cdf3a810652a"}
Oct 30 19:13:14.455319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:14.455286    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:14.481930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:14.481885    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" podStartSLOduration=2.481870635 podStartE2EDuration="2.481870635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:14.476547219 +0000 UTC m=+2414.740012187" watchObservedRunningTime="2026-10-30 19:13:14.481870635 +0000 UTC m=+2414.745335583"
Oct 30 19:13:15.467212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:15.467176    5065 generic.go:356] "Generic (PLEG): container finished" podID="1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" containerID="942f88cdd2b489068e5547b5c3c074aafd276e1995ec89009d7c680a637cd2e6" exitCode=0
Oct 30 19:13:15.467622 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:15.467243    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn" event={"ID":"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d","Type":"ContainerDied","Data":"942f88cdd2b489068e5547b5c3c074aafd276e1995ec89009d7c680a637cd2e6"}
Oct 30 19:13:16.728288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.728262    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:16.754943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.754909    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\") pod \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") "
Oct 30 19:13:16.755109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.755047    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\" (UniqueName: \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\") pod \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") "
Oct 30 19:13:16.755319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.755297    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\") pod \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\" (UID: \"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d\") "
Oct 30 19:13:16.755413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.755319    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca" pod "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" (UID: "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:16.756157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.756132    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.759053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.759027    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca" pod "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" (UID: "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:16.759431 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.759406    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp" pod "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" (UID: "1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d"). InnerVolumeSpecName "kube-api-access-2sjrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:16.858664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.858625    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjrp\" (UniqueName: \"kubernetes.io/projected/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-kube-api-access-2sjrp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.858664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.858663    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.905787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.904627    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:16.911244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.911185    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:16.921576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:16.921554    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:13:17.593111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:17.592576    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn" event={"ID":"1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d","Type":"ContainerDied","Data":"ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5"}
Oct 30 19:13:17.593111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:17.592658    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2e1c8886030b8b53f8e7c7e46d361c4d74ee928ed6c64551b63b35c8b83ab5"
Oct 30 19:13:17.593111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:17.592747    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"
Oct 30 19:13:19.455536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:19.455485    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" podUID="d4f96907-bbba-4cf5-8c22-10c98556a125" containerName="payload-processing" containerID="cri-o://28479db9dadae387264bb39236e62fda16f7b24c7c644931d54bf9cc069469a9" gracePeriod=25
Oct 30 19:13:19.617358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:19.617326    5065 generic.go:356] "Generic (PLEG): container finished" podID="d4f96907-bbba-4cf5-8c22-10c98556a125" containerID="28479db9dadae387264bb39236e62fda16f7b24c7c644931d54bf9cc069469a9" exitCode=0
Oct 30 19:13:19.617509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:19.617407    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" event={"ID":"d4f96907-bbba-4cf5-8c22-10c98556a125","Type":"ContainerDied","Data":"28479db9dadae387264bb39236e62fda16f7b24c7c644931d54bf9cc069469a9"}
Oct 30 19:13:19.914679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:19.914650    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:20.095565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.095536    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\") pod \"d4f96907-bbba-4cf5-8c22-10c98556a125\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") "
Oct 30 19:13:20.095769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.095612    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\") pod \"d4f96907-bbba-4cf5-8c22-10c98556a125\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") "
Oct 30 19:13:20.095855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.095839    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\") pod \"d4f96907-bbba-4cf5-8c22-10c98556a125\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") "
Oct 30 19:13:20.095916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.095890    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\") pod \"d4f96907-bbba-4cf5-8c22-10c98556a125\" (UID: \"d4f96907-bbba-4cf5-8c22-10c98556a125\") "
Oct 30 19:13:20.096067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.096041    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume" pod "d4f96907-bbba-4cf5-8c22-10c98556a125" (UID: "d4f96907-bbba-4cf5-8c22-10c98556a125"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:20.096325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.096303    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume" pod "d4f96907-bbba-4cf5-8c22-10c98556a125" (UID: "d4f96907-bbba-4cf5-8c22-10c98556a125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:20.096723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.096666    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.096723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.096686    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f96907-bbba-4cf5-8c22-10c98556a125-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.098312 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.098284    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials" pod "d4f96907-bbba-4cf5-8c22-10c98556a125" (UID: "d4f96907-bbba-4cf5-8c22-10c98556a125"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:20.098558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.098529    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5" pod "d4f96907-bbba-4cf5-8c22-10c98556a125" (UID: "d4f96907-bbba-4cf5-8c22-10c98556a125"). InnerVolumeSpecName "kube-api-access-5wmx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:20.198520 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.198454    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmx5\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-kube-api-access-5wmx5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.198520 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.198483    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f96907-bbba-4cf5-8c22-10c98556a125-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.630836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.630809    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"
Oct 30 19:13:20.631247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.630850    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q" event={"ID":"d4f96907-bbba-4cf5-8c22-10c98556a125","Type":"ContainerDied","Data":"aa53917cb04fc7eec0e1e5ce7e21a1a77a9a11afb4096e14c8a2cdf3a810652a"}
Oct 30 19:13:20.631247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.630915    5065 scope.go:122] "RemoveContainer" containerID="28479db9dadae387264bb39236e62fda16f7b24c7c644931d54bf9cc069469a9"
Oct 30 19:13:20.678979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.678951    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:20.690565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:20.690540    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-8cc6srn8q"]
Oct 30 19:13:21.898037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:21.897995    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f96907-bbba-4cf5-8c22-10c98556a125" path="/var/lib/kubelet/pods/d4f96907-bbba-4cf5-8c22-10c98556a125/volumes"
Oct 30 19:13:21.905766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:21.905728    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" podUID="7783e6d4-db63-480c-9be3-419910989652" containerName="payload-processing" containerID="cri-o://a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463" gracePeriod=25
Oct 30 19:13:21.922888 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:21.922851    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" podUID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" containerName="payload-processing" containerID="cri-o://1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56" gracePeriod=25
Oct 30 19:13:22.291920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.291886    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:22.335753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.335727    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:13:22.373501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373473    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\") pod \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") "
Oct 30 19:13:22.373667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373509    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\") pod \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") "
Oct 30 19:13:22.373667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373549    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\") pod \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") "
Oct 30 19:13:22.373667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\") pod \"7783e6d4-db63-480c-9be3-419910989652\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") "
Oct 30 19:13:22.373855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373787    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\") pod \"7783e6d4-db63-480c-9be3-419910989652\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") "
Oct 30 19:13:22.373855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373827    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\") pod \"7783e6d4-db63-480c-9be3-419910989652\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") "
Oct 30 19:13:22.373959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.373882    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\") pod \"7783e6d4-db63-480c-9be3-419910989652\" (UID: \"7783e6d4-db63-480c-9be3-419910989652\") "
Oct 30 19:13:22.374054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.374035    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\") pod \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\" (UID: \"5949fb9f-1086-429c-bb9d-13c952bd7b9b\") "
Oct 30 19:13:22.374487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.374464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume" pod "5949fb9f-1086-429c-bb9d-13c952bd7b9b" (UID: "5949fb9f-1086-429c-bb9d-13c952bd7b9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:22.374957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.374931    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume" pod "5949fb9f-1086-429c-bb9d-13c952bd7b9b" (UID: "5949fb9f-1086-429c-bb9d-13c952bd7b9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:22.375206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.375186    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.375292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.375211    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5949fb9f-1086-429c-bb9d-13c952bd7b9b-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.375339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.375293    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume" pod "7783e6d4-db63-480c-9be3-419910989652" (UID: "7783e6d4-db63-480c-9be3-419910989652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:22.375464 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.375443    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume" pod "7783e6d4-db63-480c-9be3-419910989652" (UID: "7783e6d4-db63-480c-9be3-419910989652"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:22.377327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.377305    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5" pod "7783e6d4-db63-480c-9be3-419910989652" (UID: "7783e6d4-db63-480c-9be3-419910989652"). InnerVolumeSpecName "kube-api-access-r6hx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.378054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.377376    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q" pod "5949fb9f-1086-429c-bb9d-13c952bd7b9b" (UID: "5949fb9f-1086-429c-bb9d-13c952bd7b9b"). InnerVolumeSpecName "kube-api-access-m272q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.378054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.377655    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials" pod "5949fb9f-1086-429c-bb9d-13c952bd7b9b" (UID: "5949fb9f-1086-429c-bb9d-13c952bd7b9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.378268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.378248    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials" pod "7783e6d4-db63-480c-9be3-419910989652" (UID: "7783e6d4-db63-480c-9be3-419910989652"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.477162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477117    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hx5\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-kube-api-access-r6hx5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.477162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477145    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7783e6d4-db63-480c-9be3-419910989652-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.477162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477155    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.477162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477164    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m272q\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-kube-api-access-m272q\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.477162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477172    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5949fb9f-1086-429c-bb9d-13c952bd7b9b-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.477528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.477182    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7783e6d4-db63-480c-9be3-419910989652-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.655852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.655814    5065 generic.go:356] "Generic (PLEG): container finished" podID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" containerID="1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56" exitCode=0
Oct 30 19:13:22.656048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.655861    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"
Oct 30 19:13:22.656048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.655903    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" event={"ID":"5949fb9f-1086-429c-bb9d-13c952bd7b9b","Type":"ContainerDied","Data":"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"}
Oct 30 19:13:22.656048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.655957    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv" event={"ID":"5949fb9f-1086-429c-bb9d-13c952bd7b9b","Type":"ContainerDied","Data":"4eadecac9947bebb169031c92c35a7e30daafb160f4855a15966dedd74bc7c29"}
Oct 30 19:13:22.656048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.655982    5065 scope.go:122] "RemoveContainer" containerID="1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"
Oct 30 19:13:22.657966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.657940    5065 generic.go:356] "Generic (PLEG): container finished" podID="7783e6d4-db63-480c-9be3-419910989652" containerID="a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463" exitCode=0
Oct 30 19:13:22.658151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.657985    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" event={"ID":"7783e6d4-db63-480c-9be3-419910989652","Type":"ContainerDied","Data":"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"}
Oct 30 19:13:22.658151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.657995    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"
Oct 30 19:13:22.658151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.658024    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq" event={"ID":"7783e6d4-db63-480c-9be3-419910989652","Type":"ContainerDied","Data":"b87825783e0119952f9e4f51e0ee252af00186d0ef98b9a6cebb3c89feac89f3"}
Oct 30 19:13:22.669681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.669666    5065 scope.go:122] "RemoveContainer" containerID="1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"
Oct 30 19:13:22.669962 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:22.669939    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56\": container with ID starting with 1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56 not found: ID does not exist" containerID="1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"
Oct 30 19:13:22.670018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.669970    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56"} err="failed to get container status \"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56\": rpc error: code = NotFound desc = could not find container \"1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56\": container with ID starting with 1d734886a644daafde5175f14f0d27adbf992f79dc9df43032a3c9d6e4feaa56 not found: ID does not exist"
Oct 30 19:13:22.670018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.669986    5065 scope.go:122] "RemoveContainer" containerID="a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"
Oct 30 19:13:22.681236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.681216    5065 scope.go:122] "RemoveContainer" containerID="a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"
Oct 30 19:13:22.681571 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:22.681542    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463\": container with ID starting with a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463 not found: ID does not exist" containerID="a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"
Oct 30 19:13:22.681671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.681581    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463"} err="failed to get container status \"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463\": rpc error: code = NotFound desc = could not find container \"a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463\": container with ID starting with a6501c8b22b89df8ba5ba348065244b7126f8edd0dba32396946c77f689d5463 not found: ID does not exist"
Oct 30 19:13:22.697204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.697180    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:13:22.701758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.701736    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-bfd9qnstv"]
Oct 30 19:13:22.994519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:22.986598    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:23.033110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:23.021234    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-6f45vrtcq"]
Oct 30 19:13:23.152536 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:23.152495    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:13:23.896018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:23.895981    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" path="/var/lib/kubelet/pods/5949fb9f-1086-429c-bb9d-13c952bd7b9b/volumes"
Oct 30 19:13:23.897926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:23.897896    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7783e6d4-db63-480c-9be3-419910989652" path="/var/lib/kubelet/pods/7783e6d4-db63-480c-9be3-419910989652/volumes"
Oct 30 19:13:25.533148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.533111    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"]
Oct 30 19:13:25.533989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.533971    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" containerName="payload-processing"
Oct 30 19:13:25.533989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.533987    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" containerName="payload-processing"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534004    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7783e6d4-db63-480c-9be3-419910989652" containerName="payload-processing"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534009    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7783e6d4-db63-480c-9be3-419910989652" containerName="payload-processing"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534025    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" containerName="revoke-keys"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534031    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" containerName="revoke-keys"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534044    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d4f96907-bbba-4cf5-8c22-10c98556a125" containerName="payload-processing"
Oct 30 19:13:25.534221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534049    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f96907-bbba-4cf5-8c22-10c98556a125" containerName="payload-processing"
Oct 30 19:13:25.534563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534266    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5949fb9f-1086-429c-bb9d-13c952bd7b9b" containerName="payload-processing"
Oct 30 19:13:25.534563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534280    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7783e6d4-db63-480c-9be3-419910989652" containerName="payload-processing"
Oct 30 19:13:25.534563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534290    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" containerName="revoke-keys"
Oct 30 19:13:25.534563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.534301    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f96907-bbba-4cf5-8c22-10c98556a125" containerName="payload-processing"
Oct 30 19:13:25.538563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.538538    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.551591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.551560    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"kube-root-ca.crt\""
Oct 30 19:13:25.551591 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.551572    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"openshift-service-ca.crt\""
Oct 30 19:13:25.553641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.553604    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"routing-overlay\""
Oct 30 19:13:25.553813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.553735    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8c554\""
Oct 30 19:13:25.562003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.561968    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"agc-cred-ffbcd05a91cf\""
Oct 30 19:13:25.562152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.562134    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-e2e--307b9dfd-dockercfg-kgnks\""
Oct 30 19:13:25.562518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.562496    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"]
Oct 30 19:13:25.601654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.601624    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.601847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.601803    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.601924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.601870    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.601924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.601911    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzw6\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.617716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.617677    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"]
Oct 30 19:13:25.618377 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.618344    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dgzw6 plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" podUID="e584081c-ec05-4e16-ac4b-212ea1a04cbb"
Oct 30 19:13:25.703204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.703173    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.703437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.703260    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.703437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.703302    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzw6\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.703437 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.703405    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:25.703437 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.703426    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:25.703691 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.703483    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:26.203465756 +0000 UTC m=+2426.466930691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:25.703691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.703613    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.703822 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.703795    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:13:25.703879 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.703858    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:26.203839391 +0000 UTC m=+2426.467304332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:13:25.703959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.703939    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:25.712503 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.712486    5065 projected.go:196] Error preparing data for projected volume kube-api-access-dgzw6 for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e8c554" not found
Oct 30 19:13:25.712621 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:25.712584    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6 podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:26.212525961 +0000 UTC m=+2426.475990888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzw6" (UniqueName: "kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e8c554" not found
Oct 30 19:13:25.943169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:25.935525    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:26.100500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.100468    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:26.253963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.253880    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\") pod \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") "
Oct 30 19:13:26.254284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.254260    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume" pod "e584081c-ec05-4e16-ac4b-212ea1a04cbb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:26.254438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.254420    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:26.254504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.254456    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzw6\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:26.254597 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.254580    5065 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e8c554/agc-cred-ffbcd05a91cf: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:26.254663 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.254602    5065 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb: secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:26.254720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.254656    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb\" (UID: \"e584081c-ec05-4e16-ac4b-212ea1a04cbb\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:26.254720 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.254676    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:27.254655041 +0000 UTC m=+2427.518119982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : secret "agc-cred-ffbcd05a91cf" not found
Oct 30 19:13:26.254720 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.254703    5065 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-plugins-e2e-extmod-a-e8c554: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:13:26.254839 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.254751    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:27.254736592 +0000 UTC m=+2427.518201518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e8c554" not found
Oct 30 19:13:26.254883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.254870    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:26.264449 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.264416    5065 projected.go:196] Error preparing data for projected volume kube-api-access-dgzw6 for pod ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e8c554" not found
Oct 30 19:13:26.264597 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:26.264528    5065 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6 podName:e584081c-ec05-4e16-ac4b-212ea1a04cbb nodeName:}" failed. No retries permitted until 2026-10-30 19:13:27.26449723 +0000 UTC m=+2427.527962156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzw6" (UniqueName: "kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6") pod "payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb" (UID: "e584081c-ec05-4e16-ac4b-212ea1a04cbb") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-e8c554" not found
Oct 30 19:13:26.946477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:26.946447    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"
Oct 30 19:13:27.023164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.023129    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"]
Oct 30 19:13:27.034626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.034595    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-7b78zd9zb"]
Oct 30 19:13:27.070927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.070899    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:27.070927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.070923    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzw6\" (UniqueName: \"kubernetes.io/projected/e584081c-ec05-4e16-ac4b-212ea1a04cbb-kube-api-access-dgzw6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:27.070927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.070934    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e584081c-ec05-4e16-ac4b-212ea1a04cbb-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:27.892708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:27.892678    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e584081c-ec05-4e16-ac4b-212ea1a04cbb" path="/var/lib/kubelet/pods/e584081c-ec05-4e16-ac4b-212ea1a04cbb/volumes"
Oct 30 19:13:30.703118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.703054    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:30.709102 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.709064    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.720561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.720540    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ac4d34ebc773\""
Oct 30 19:13:30.721239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.721218    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 19:13:30.721380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.721231    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 19:13:30.721461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.721245    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wclh5\""
Oct 30 19:13:30.721461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.721259    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 19:13:30.722504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.722487    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 19:13:30.722716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.722699    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:30.800975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.800450    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.800975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.800624    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjxr\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.800975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.800902    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.800975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.800984    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.911112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.904359    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.911112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.904511    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjxr\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.911112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.904683    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.911112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.904753    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.942953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.942920    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.965432 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.944528    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.965681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.948780    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:30.965878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:30.965031    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjxr\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\") pod \"payload-processing-external-model-7858c7b74f-47xdf\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") " pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:31.040458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:31.040418    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:31.686150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:31.686119    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:32.120240 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:32.120194    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" event={"ID":"34485408-87aa-4943-8076-f54a2b32ff30","Type":"ContainerStarted","Data":"d61e609110103bb687e5ae15ba9a02ca18fda8223b074294d918c3da3ded9122"}
Oct 30 19:13:32.120892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:32.120848    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" event={"ID":"34485408-87aa-4943-8076-f54a2b32ff30","Type":"ContainerStarted","Data":"e30dedd465de9932866ab5dd5bbdcd97de728fb4729580a2ffdb1e30970852dd"}
Oct 30 19:13:32.122693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:32.122671    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:32.173199 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:32.173071    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" podStartSLOduration=2.173056305 podStartE2EDuration="2.173056305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:32.166698808 +0000 UTC m=+2432.430163757" watchObservedRunningTime="2026-10-30 19:13:32.173056305 +0000 UTC m=+2432.436521293"
Oct 30 19:13:34.558813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.558781    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:13:34.564577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.564553    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.574022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.573993    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fee8e82c3bd8\""
Oct 30 19:13:34.579105 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.579059    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:13:34.706416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.706379    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.706632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.706615    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.706693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.706681    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.706891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.706866    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.808356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.808320    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.808543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.808427    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.808543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.808528    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.808720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.808705    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.809256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.809178    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.809544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.809300    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.811388 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.811360    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.823331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.823309    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\") pod \"payload-processing-external-model-8675cb7d68-kptrc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") " pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:34.901099 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:34.901062    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:35.201418 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:35.201390    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473e0675_24e9_4845_9e58_bf5e7624b0bc.slice/crio-65df40f8f11dc3d7ed2388b387133c0a79f976c358090383d990df9722bb9db2 WatchSource:0}: Error finding container 65df40f8f11dc3d7ed2388b387133c0a79f976c358090383d990df9722bb9db2: Status 404 returned error can't find the container with id 65df40f8f11dc3d7ed2388b387133c0a79f976c358090383d990df9722bb9db2
Oct 30 19:13:35.203926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:35.203899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:13:35.285177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:35.285144    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" event={"ID":"473e0675-24e9-4845-9e58-bf5e7624b0bc","Type":"ContainerStarted","Data":"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"}
Oct 30 19:13:35.285369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:35.285203    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" event={"ID":"473e0675-24e9-4845-9e58-bf5e7624b0bc","Type":"ContainerStarted","Data":"65df40f8f11dc3d7ed2388b387133c0a79f976c358090383d990df9722bb9db2"}
Oct 30 19:13:35.285369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:35.285245    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:35.311771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:35.311722    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" podStartSLOduration=1.3117076189999999 podStartE2EDuration="1.311707619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:35.308668363 +0000 UTC m=+2435.572133311" watchObservedRunningTime="2026-10-30 19:13:35.311707619 +0000 UTC m=+2435.575172566"
Oct 30 19:13:42.584963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.584921    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:42.588573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.588549    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:42.626537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.626502    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:13:42.633713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.633686    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.640122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.640062    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:13:42.640737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.640714    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8348b0e5ef0b\""
Oct 30 19:13:42.735416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.735374    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9f5d\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.735689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.735459    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.735689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.735637    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.735863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.735762    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.838062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.837974    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9f5d\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.838062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.838025    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.838333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.838190    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.838333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.838305    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.838776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.838751    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.839071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.839047    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.841025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.841005    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.854114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.854071    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9f5d\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\") pod \"payload-processing-external-model-85ffd45dc8-tpp8p\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") " pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:42.962578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:42.962552    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:43.335129 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:43.330041    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f3267_6025_4f61_8207_3bdea2b85bee.slice/crio-dd52625dff05099371306f59eee477f26a664672b3505240e32355419724b517 WatchSource:0}: Error finding container dd52625dff05099371306f59eee477f26a664672b3505240e32355419724b517: Status 404 returned error can't find the container with id dd52625dff05099371306f59eee477f26a664672b3505240e32355419724b517
Oct 30 19:13:43.338521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:43.338449    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:13:43.475338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:43.475302    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" event={"ID":"8e5f3267-6025-4f61-8207-3bdea2b85bee","Type":"ContainerStarted","Data":"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"}
Oct 30 19:13:43.475493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:43.475367    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:43.475493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:43.475389    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" event={"ID":"8e5f3267-6025-4f61-8207-3bdea2b85bee","Type":"ContainerStarted","Data":"dd52625dff05099371306f59eee477f26a664672b3505240e32355419724b517"}
Oct 30 19:13:43.499036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:43.498989    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" podStartSLOduration=1.498975456 podStartE2EDuration="1.498975456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:43.496762343 +0000 UTC m=+2443.760227304" watchObservedRunningTime="2026-10-30 19:13:43.498975456 +0000 UTC m=+2443.762440405"
Oct 30 19:13:46.300926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:46.300894    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:13:47.593220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:47.591723    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" podUID="34485408-87aa-4943-8076-f54a2b32ff30" containerName="payload-processing" containerID="cri-o://d61e609110103bb687e5ae15ba9a02ca18fda8223b074294d918c3da3ded9122" gracePeriod=25
Oct 30 19:13:48.800699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.776918    5065 generic.go:356] "Generic (PLEG): container finished" podID="34485408-87aa-4943-8076-f54a2b32ff30" containerID="d61e609110103bb687e5ae15ba9a02ca18fda8223b074294d918c3da3ded9122" exitCode=0
Oct 30 19:13:48.800699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.777261    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" event={"ID":"34485408-87aa-4943-8076-f54a2b32ff30","Type":"ContainerDied","Data":"d61e609110103bb687e5ae15ba9a02ca18fda8223b074294d918c3da3ded9122"}
Oct 30 19:13:48.851690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.851235    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.916948    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\") pod \"34485408-87aa-4943-8076-f54a2b32ff30\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") "
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.917599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume" pod "34485408-87aa-4943-8076-f54a2b32ff30" (UID: "34485408-87aa-4943-8076-f54a2b32ff30"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.917664    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\") pod \"34485408-87aa-4943-8076-f54a2b32ff30\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") "
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.917882    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\") pod \"34485408-87aa-4943-8076-f54a2b32ff30\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") "
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.917979    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\") pod \"34485408-87aa-4943-8076-f54a2b32ff30\" (UID: \"34485408-87aa-4943-8076-f54a2b32ff30\") "
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.919317    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:48.921386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.921345    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume" pod "34485408-87aa-4943-8076-f54a2b32ff30" (UID: "34485408-87aa-4943-8076-f54a2b32ff30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:48.960048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.956718    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr" pod "34485408-87aa-4943-8076-f54a2b32ff30" (UID: "34485408-87aa-4943-8076-f54a2b32ff30"). InnerVolumeSpecName "kube-api-access-xrjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:48.960048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:48.959745    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials" pod "34485408-87aa-4943-8076-f54a2b32ff30" (UID: "34485408-87aa-4943-8076-f54a2b32ff30"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:49.022149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.022031    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:49.022149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.022076    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34485408-87aa-4943-8076-f54a2b32ff30-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:49.022149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.022121    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjxr\" (UniqueName: \"kubernetes.io/projected/34485408-87aa-4943-8076-f54a2b32ff30-kube-api-access-xrjxr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:49.858114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.857844    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"
Oct 30 19:13:49.858634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.858186    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf" event={"ID":"34485408-87aa-4943-8076-f54a2b32ff30","Type":"ContainerDied","Data":"e30dedd465de9932866ab5dd5bbdcd97de728fb4729580a2ffdb1e30970852dd"}
Oct 30 19:13:49.858634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.858251    5065 scope.go:122] "RemoveContainer" containerID="d61e609110103bb687e5ae15ba9a02ca18fda8223b074294d918c3da3ded9122"
Oct 30 19:13:49.862749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.862721    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"]
Oct 30 19:13:49.899287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.891301    5065 cpu_manager.go:414] "containerMap: removing container" podUID="34485408-87aa-4943-8076-f54a2b32ff30" containerName="payload-processing"
Oct 30 19:13:49.899287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.891329    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34485408-87aa-4943-8076-f54a2b32ff30" containerName="payload-processing"
Oct 30 19:13:49.899287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.891967    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="34485408-87aa-4943-8076-f54a2b32ff30" containerName="payload-processing"
Oct 30 19:13:49.967178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.955103    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"]
Oct 30 19:13:49.967178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:49.957007    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.029293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.029126    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:50.029641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.029624    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:13:50.117163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.111845    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.117163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.111966    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.117163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.112014    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtzpw\" (UniqueName: \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.238117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.238058    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.238449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.238425    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.238607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.238589    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtzpw\" (UniqueName: \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.239859 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.239834    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.246498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.246471    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.259241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.259200    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:50.262744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.262713    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtzpw\" (UniqueName: \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\") pod \"maas-api-revoke-sub-53bed78f-lf6k9\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.268890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.266481    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7858c7b74f-47xdf"]
Oct 30 19:13:50.333637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.333139    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:50.766200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.766159    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 19:13:50.766646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.766617    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" containerID="cri-o://79813a290fdebb9c8eb745ea26fdae4f5d8b80d8e7c5b4d6842cb7374b90e937" gracePeriod=30
Oct 30 19:13:50.777601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:50.777576    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"]
Oct 30 19:13:51.020181 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:50.992976    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4cd31c_d1fe_4921_a1e6_9daad77f0731.slice/crio-2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00 WatchSource:0}: Error finding container 2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00: Status 404 returned error can't find the container with id 2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00
Oct 30 19:13:51.111809 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.111752    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"]
Oct 30 19:13:51.159411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.159380    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1078e303-xhcmd"]
Oct 30 19:13:51.212111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.211477    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"]
Oct 30 19:13:51.238350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.225080    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"]
Oct 30 19:13:51.238350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.225227    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.325167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.322171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.325167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.322418    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk78j\" (UniqueName: \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.329387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.322515    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.431062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.431028    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk78j\" (UniqueName: \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.431252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.431210    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.431488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.431469    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.432106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.432037    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.433965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.433943    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.444562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.444539    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk78j\" (UniqueName: \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\") pod \"maas-api-revoke-sub-fe1dba16-xms97\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.572499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.572398    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:51.875787 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:13:51.875758    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c056dd9_a767_4342_b567_18b7ac210ef9.slice/crio-22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3 WatchSource:0}: Error finding container 22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3: Status 404 returned error can't find the container with id 22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3
Oct 30 19:13:51.877882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.877852    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"]
Oct 30 19:13:51.906542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.906510    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34485408-87aa-4943-8076-f54a2b32ff30" path="/var/lib/kubelet/pods/34485408-87aa-4943-8076-f54a2b32ff30/volumes"
Oct 30 19:13:51.907077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:51.907058    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c378a181-b022-4b5b-bebc-3f755f257cfe" path="/var/lib/kubelet/pods/c378a181-b022-4b5b-bebc-3f755f257cfe/volumes"
Oct 30 19:13:52.114555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:52.114513    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9" event={"ID":"7b4cd31c-d1fe-4921-a1e6-9daad77f0731","Type":"ContainerStarted","Data":"9eb3e7f30b34288425af33deaf9769f3175e94c2536f61e0e4a7aa9734766812"}
Oct 30 19:13:52.115627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:52.114585    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9" event={"ID":"7b4cd31c-d1fe-4921-a1e6-9daad77f0731","Type":"ContainerStarted","Data":"2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00"}
Oct 30 19:13:52.116423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:52.116394    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97" event={"ID":"5c056dd9-a767-4342-b567-18b7ac210ef9","Type":"ContainerStarted","Data":"22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3"}
Oct 30 19:13:52.139864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:52.139787    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9" podStartSLOduration=3.139774227 podStartE2EDuration="3.139774227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:52.135125457 +0000 UTC m=+2452.398590406" watchObservedRunningTime="2026-10-30 19:13:52.139774227 +0000 UTC m=+2452.403239174"
Oct 30 19:13:53.135621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:53.135587    5065 generic.go:356] "Generic (PLEG): container finished" podID="7b4cd31c-d1fe-4921-a1e6-9daad77f0731" containerID="9eb3e7f30b34288425af33deaf9769f3175e94c2536f61e0e4a7aa9734766812" exitCode=0
Oct 30 19:13:53.136131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:53.135839    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9" event={"ID":"7b4cd31c-d1fe-4921-a1e6-9daad77f0731","Type":"ContainerDied","Data":"9eb3e7f30b34288425af33deaf9769f3175e94c2536f61e0e4a7aa9734766812"}
Oct 30 19:13:53.138393 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:13:53.138359    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:13:53.138606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:53.138586    5065 generic.go:356] "Generic (PLEG): container finished" podID="5c056dd9-a767-4342-b567-18b7ac210ef9" containerID="2f6cfb6aac5c971c44c2c4009587e6e43407a2542e8db28d2b9f0681225c68d4" exitCode=0
Oct 30 19:13:53.138693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:53.138673    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97" event={"ID":"5c056dd9-a767-4342-b567-18b7ac210ef9","Type":"ContainerDied","Data":"2f6cfb6aac5c971c44c2c4009587e6e43407a2542e8db28d2b9f0681225c68d4"}
Oct 30 19:13:54.402529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.402465    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:54.407254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.407231    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:54.489412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.489380    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:13:54.496166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496128    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\" (UniqueName: \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\") pod \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") "
Oct 30 19:13:54.496331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496278    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\") pod \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") "
Oct 30 19:13:54.496529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496500    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\" (UniqueName: \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\") pod \"5c056dd9-a767-4342-b567-18b7ac210ef9\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") "
Oct 30 19:13:54.496672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496655    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\") pod \"5c056dd9-a767-4342-b567-18b7ac210ef9\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") "
Oct 30 19:13:54.496795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496786    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\") pod \"5c056dd9-a767-4342-b567-18b7ac210ef9\" (UID: \"5c056dd9-a767-4342-b567-18b7ac210ef9\") "
Oct 30 19:13:54.496864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.496832    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\") pod \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\" (UID: \"7b4cd31c-d1fe-4921-a1e6-9daad77f0731\") "
Oct 30 19:13:54.497777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.497497    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca" pod "5c056dd9-a767-4342-b567-18b7ac210ef9" (UID: "5c056dd9-a767-4342-b567-18b7ac210ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:54.498208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.498065    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.498524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.498498    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca" pod "7b4cd31c-d1fe-4921-a1e6-9daad77f0731" (UID: "7b4cd31c-d1fe-4921-a1e6-9daad77f0731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:54.499453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.499428    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw" pod "7b4cd31c-d1fe-4921-a1e6-9daad77f0731" (UID: "7b4cd31c-d1fe-4921-a1e6-9daad77f0731"). InnerVolumeSpecName "kube-api-access-mtzpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:54.501396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.501366    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca" pod "5c056dd9-a767-4342-b567-18b7ac210ef9" (UID: "5c056dd9-a767-4342-b567-18b7ac210ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:54.501689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.501664    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j" pod "5c056dd9-a767-4342-b567-18b7ac210ef9" (UID: "5c056dd9-a767-4342-b567-18b7ac210ef9"). InnerVolumeSpecName "kube-api-access-bk78j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:54.502048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.502020    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca" pod "7b4cd31c-d1fe-4921-a1e6-9daad77f0731" (UID: "7b4cd31c-d1fe-4921-a1e6-9daad77f0731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:54.553149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.553117    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:13:54.600746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.600716    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c056dd9-a767-4342-b567-18b7ac210ef9-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.600975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.600949    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.600975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.600979    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtzpw\" (UniqueName: \"kubernetes.io/projected/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-kube-api-access-mtzpw\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.601195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.600994    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cd31c-d1fe-4921-a1e6-9daad77f0731-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:54.601195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:54.601008    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk78j\" (UniqueName: \"kubernetes.io/projected/5c056dd9-a767-4342-b567-18b7ac210ef9-kube-api-access-bk78j\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:55.161244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.161211    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"
Oct 30 19:13:55.161244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.161231    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97" event={"ID":"5c056dd9-a767-4342-b567-18b7ac210ef9","Type":"ContainerDied","Data":"22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3"}
Oct 30 19:13:55.161514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.161318    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22e790aa5b529bbc49dae1b8fd3f45719773570666a49f4ec07ec0f7d7763fa3"
Oct 30 19:13:55.163168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.163144    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"
Oct 30 19:13:55.163168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.163156    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9" event={"ID":"7b4cd31c-d1fe-4921-a1e6-9daad77f0731","Type":"ContainerDied","Data":"2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00"}
Oct 30 19:13:55.163379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.163202    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b846a3de9b69b7258ce1f11a81142ec021da107d176409d40783ad2532bfe00"
Oct 30 19:13:55.804340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.804303    5065 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:55.804809 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:55.804384    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:56.177599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.177566    5065 generic.go:356] "Generic (PLEG): container finished" podID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerID="79813a290fdebb9c8eb745ea26fdae4f5d8b80d8e7c5b4d6842cb7374b90e937" exitCode=0
Oct 30 19:13:56.177789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.177636    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" event={"ID":"1caf1e42-14fd-432b-b3d6-164c7d02ec66","Type":"ContainerDied","Data":"79813a290fdebb9c8eb745ea26fdae4f5d8b80d8e7c5b4d6842cb7374b90e937"}
Oct 30 19:13:56.276437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.276415    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 19:13:56.355870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.355787    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.355965    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356172    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356216    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356371    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356405    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356437    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.356576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356443    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356604    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.356798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.356713    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\") pod \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\" (UID: \"1caf1e42-14fd-432b-b3d6-164c7d02ec66\") "
Oct 30 19:13:56.358047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.357396    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.358047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.357610    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.358047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.357826    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.358278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358202    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.358278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358228    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.358278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358244    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.358278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358258    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.358606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358487    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:56.358781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.358759    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:56.359530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.359509    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "kube-api-access-h6bcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:56.360302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.360285    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:56.360548 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.360527    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy" pod "1caf1e42-14fd-432b-b3d6-164c7d02ec66" (UID: "1caf1e42-14fd-432b-b3d6-164c7d02ec66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.459524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.459494    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.459524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.459523    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6bcm\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-kube-api-access-h6bcm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.459734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.459533    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.459734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.459543    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.459734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:56.459553    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1caf1e42-14fd-432b-b3d6-164c7d02ec66-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.191019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.190983    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"
Oct 30 19:13:57.191019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.191006    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v" event={"ID":"1caf1e42-14fd-432b-b3d6-164c7d02ec66","Type":"ContainerDied","Data":"f91ebb438eefe09a2d81e2a0e4534b872dce8824461100fd9a59d8aceb567fce"}
Oct 30 19:13:57.191633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.191070    5065 scope.go:122] "RemoveContainer" containerID="79813a290fdebb9c8eb745ea26fdae4f5d8b80d8e7c5b4d6842cb7374b90e937"
Oct 30 19:13:57.225459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.225425    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 19:13:57.231441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.231419    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hnr2v"]
Oct 30 19:13:57.918824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:57.918787    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" path="/var/lib/kubelet/pods/1caf1e42-14fd-432b-b3d6-164c7d02ec66/volumes"
Oct 30 19:13:59.554516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:13:59.554448    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" podUID="473e0675-24e9-4845-9e58-bf5e7624b0bc" containerName="payload-processing" containerID="cri-o://0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef" gracePeriod=25
Oct 30 19:14:00.000236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.000208    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:14:00.146679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.146350    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\") pod \"473e0675-24e9-4845-9e58-bf5e7624b0bc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") "
Oct 30 19:14:00.146679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.146600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\") pod \"473e0675-24e9-4845-9e58-bf5e7624b0bc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") "
Oct 30 19:14:00.146992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.146876    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\") pod \"473e0675-24e9-4845-9e58-bf5e7624b0bc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") "
Oct 30 19:14:00.146992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.146986    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\") pod \"473e0675-24e9-4845-9e58-bf5e7624b0bc\" (UID: \"473e0675-24e9-4845-9e58-bf5e7624b0bc\") "
Oct 30 19:14:00.149751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.149278    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume" pod "473e0675-24e9-4845-9e58-bf5e7624b0bc" (UID: "473e0675-24e9-4845-9e58-bf5e7624b0bc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:00.149751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.149481    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume" pod "473e0675-24e9-4845-9e58-bf5e7624b0bc" (UID: "473e0675-24e9-4845-9e58-bf5e7624b0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:00.150072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.150054    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.150191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.150075    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e0675-24e9-4845-9e58-bf5e7624b0bc-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.151494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.151474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm" pod "473e0675-24e9-4845-9e58-bf5e7624b0bc" (UID: "473e0675-24e9-4845-9e58-bf5e7624b0bc"). InnerVolumeSpecName "kube-api-access-dchfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:00.151638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.151600    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials" pod "473e0675-24e9-4845-9e58-bf5e7624b0bc" (UID: "473e0675-24e9-4845-9e58-bf5e7624b0bc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:00.230794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.230726    5065 generic.go:356] "Generic (PLEG): container finished" podID="473e0675-24e9-4845-9e58-bf5e7624b0bc" containerID="0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef" exitCode=0
Oct 30 19:14:00.230912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.230793    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"
Oct 30 19:14:00.230912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.230808    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" event={"ID":"473e0675-24e9-4845-9e58-bf5e7624b0bc","Type":"ContainerDied","Data":"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"}
Oct 30 19:14:00.230912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.230892    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc" event={"ID":"473e0675-24e9-4845-9e58-bf5e7624b0bc","Type":"ContainerDied","Data":"65df40f8f11dc3d7ed2388b387133c0a79f976c358090383d990df9722bb9db2"}
Oct 30 19:14:00.231021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.230920    5065 scope.go:122] "RemoveContainer" containerID="0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"
Oct 30 19:14:00.243407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.243389    5065 scope.go:122] "RemoveContainer" containerID="0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"
Oct 30 19:14:00.243710 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:00.243691    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef\": container with ID starting with 0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef not found: ID does not exist" containerID="0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"
Oct 30 19:14:00.243765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.243720    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef"} err="failed to get container status \"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef\": rpc error: code = NotFound desc = could not find container \"0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef\": container with ID starting with 0d125614e33d8753123c2057b84703998bb9b7d3ca877ff0032bb7d175d952ef not found: ID does not exist"
Oct 30 19:14:00.251450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.251428    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchfm\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-kube-api-access-dchfm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.251553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.251453    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/473e0675-24e9-4845-9e58-bf5e7624b0bc-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.261879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.261858    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:14:00.267410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.267390    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675cb7d68-kptrc"]
Oct 30 19:14:00.432141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.432105    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"]
Oct 30 19:14:00.433636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433607    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy"
Oct 30 19:14:00.433636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433636    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433668    5065 cpu_manager.go:414] "containerMap: removing container" podUID="473e0675-24e9-4845-9e58-bf5e7624b0bc" containerName="payload-processing"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433677    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473e0675-24e9-4845-9e58-bf5e7624b0bc" containerName="payload-processing"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433733    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7b4cd31c-d1fe-4921-a1e6-9daad77f0731" containerName="revoke-keys"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433741    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4cd31c-d1fe-4921-a1e6-9daad77f0731" containerName="revoke-keys"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433806    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5c056dd9-a767-4342-b567-18b7ac210ef9" containerName="revoke-keys"
Oct 30 19:14:00.433850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.433814    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c056dd9-a767-4342-b567-18b7ac210ef9" containerName="revoke-keys"
Oct 30 19:14:00.434347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.434020    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4cd31c-d1fe-4921-a1e6-9daad77f0731" containerName="revoke-keys"
Oct 30 19:14:00.434347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.434042    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c056dd9-a767-4342-b567-18b7ac210ef9" containerName="revoke-keys"
Oct 30 19:14:00.434347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.434054    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1caf1e42-14fd-432b-b3d6-164c7d02ec66" containerName="istio-proxy"
Oct 30 19:14:00.434347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.434065    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="473e0675-24e9-4845-9e58-bf5e7624b0bc" containerName="payload-processing"
Oct 30 19:14:00.440914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.440890    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.442661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.442637    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"]
Oct 30 19:14:00.445545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.445525    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:00.445810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.445794    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:14:00.557320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.557285    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.557743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.557556    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks92z\" (UniqueName: \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.557743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.557633    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.659569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.659537    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.659751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.659706    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks92z\" (UniqueName: \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.659821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.659757    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.660613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.660589    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.662826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.662798    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.674130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.674102    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks92z\" (UniqueName: \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\") pod \"maas-api-revoke-sub-e8c27ed4-4bjhg\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:00.770106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:00.767562    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:01.163772 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:14:01.163745    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dff3ce_aaca_4bad_a0d6_bfad13e2cfff.slice/crio-731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101 WatchSource:0}: Error finding container 731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101: Status 404 returned error can't find the container with id 731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101
Oct 30 19:14:01.166168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:01.166144    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"]
Oct 30 19:14:01.244143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:01.244114    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg" event={"ID":"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff","Type":"ContainerStarted","Data":"731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101"}
Oct 30 19:14:01.895424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:01.895392    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473e0675-24e9-4845-9e58-bf5e7624b0bc" path="/var/lib/kubelet/pods/473e0675-24e9-4845-9e58-bf5e7624b0bc/volumes"
Oct 30 19:14:02.257976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:02.257888    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg" event={"ID":"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff","Type":"ContainerStarted","Data":"cad8a78d92b72b7f6d1f2d3f9d01d5e89fc754ad41b50a42601fb285e5f393d1"}
Oct 30 19:14:02.285183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:02.285116    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg" podStartSLOduration=2.285078635 podStartE2EDuration="2.285078635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:02.281519436 +0000 UTC m=+2462.544984385" watchObservedRunningTime="2026-10-30 19:14:02.285078635 +0000 UTC m=+2462.548543584"
Oct 30 19:14:02.330907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:02.330876    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:14:03.269952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:03.269920    5065 generic.go:356] "Generic (PLEG): container finished" podID="97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" containerID="cad8a78d92b72b7f6d1f2d3f9d01d5e89fc754ad41b50a42601fb285e5f393d1" exitCode=0
Oct 30 19:14:03.270469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:03.270010    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg" event={"ID":"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff","Type":"ContainerDied","Data":"cad8a78d92b72b7f6d1f2d3f9d01d5e89fc754ad41b50a42601fb285e5f393d1"}
Oct 30 19:14:04.481738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.481711    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:04.585597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.585495    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\") pod \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") "
Oct 30 19:14:04.585788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.585595    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\" (UniqueName: \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\") pod \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") "
Oct 30 19:14:04.585875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.585833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\") pod \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\" (UID: \"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff\") "
Oct 30 19:14:04.586481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.586447    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca" pod "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" (UID: "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:04.586923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.586893    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:04.588816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.588773    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z" pod "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" (UID: "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff"). InnerVolumeSpecName "kube-api-access-ks92z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:04.589267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.589240    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca" pod "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" (UID: "97dff3ce-aaca-4bad-a0d6-bfad13e2cfff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:04.688372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.688340    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:04.688372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:04.688369    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks92z\" (UniqueName: \"kubernetes.io/projected/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff-kube-api-access-ks92z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:05.293415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:05.293388    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"
Oct 30 19:14:05.293583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:05.293381    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg" event={"ID":"97dff3ce-aaca-4bad-a0d6-bfad13e2cfff","Type":"ContainerDied","Data":"731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101"}
Oct 30 19:14:05.293583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:05.293519    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="731637f199674f3d385ac8a08e58adccf11d0b760682a6d204bab9a0d9fa9101"
Oct 30 19:14:07.332073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.331995    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" podUID="53022857-f972-44e2-8aa4-4aae0700fc8e" containerName="payload-processing" containerID="cri-o://5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe" gracePeriod=25
Oct 30 19:14:07.639554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.639524    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:14:07.750139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.750103    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\") pod \"53022857-f972-44e2-8aa4-4aae0700fc8e\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") "
Oct 30 19:14:07.750335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.750296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\") pod \"53022857-f972-44e2-8aa4-4aae0700fc8e\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") "
Oct 30 19:14:07.750450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.750436    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\") pod \"53022857-f972-44e2-8aa4-4aae0700fc8e\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") "
Oct 30 19:14:07.750510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.750460    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume" pod "53022857-f972-44e2-8aa4-4aae0700fc8e" (UID: "53022857-f972-44e2-8aa4-4aae0700fc8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:07.750573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.750548    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\") pod \"53022857-f972-44e2-8aa4-4aae0700fc8e\" (UID: \"53022857-f972-44e2-8aa4-4aae0700fc8e\") "
Oct 30 19:14:07.751165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.751144    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.751289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.751206    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume" pod "53022857-f972-44e2-8aa4-4aae0700fc8e" (UID: "53022857-f972-44e2-8aa4-4aae0700fc8e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:07.752707 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.752682    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials" pod "53022857-f972-44e2-8aa4-4aae0700fc8e" (UID: "53022857-f972-44e2-8aa4-4aae0700fc8e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:07.752803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.752720    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2" pod "53022857-f972-44e2-8aa4-4aae0700fc8e" (UID: "53022857-f972-44e2-8aa4-4aae0700fc8e"). InnerVolumeSpecName "kube-api-access-mpxt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:07.853073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.853001    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxt2\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-kube-api-access-mpxt2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.853073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.853033    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53022857-f972-44e2-8aa4-4aae0700fc8e-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.853073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:07.853048    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53022857-f972-44e2-8aa4-4aae0700fc8e-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:08.189966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.189876    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 19:14:08.190241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.190153    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy" containerID="cri-o://b44e4e6ec20240115d2254c9374d694bf6bfae83dce1dc7e0d5e3113ac8ef9ed" gracePeriod=30
Oct 30 19:14:08.335279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.335240    5065 generic.go:356] "Generic (PLEG): container finished" podID="53022857-f972-44e2-8aa4-4aae0700fc8e" containerID="5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe" exitCode=0
Oct 30 19:14:08.335797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.335305    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"
Oct 30 19:14:08.335797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.335339    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" event={"ID":"53022857-f972-44e2-8aa4-4aae0700fc8e","Type":"ContainerDied","Data":"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"}
Oct 30 19:14:08.335797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.335418    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm" event={"ID":"53022857-f972-44e2-8aa4-4aae0700fc8e","Type":"ContainerDied","Data":"545be99c36a1597567bcc8c4357439a45de9b0ccdf17b156eceab1eb609dd916"}
Oct 30 19:14:08.335797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.335454    5065 scope.go:122] "RemoveContainer" containerID="5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"
Oct 30 19:14:08.349046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.349029    5065 scope.go:122] "RemoveContainer" containerID="5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"
Oct 30 19:14:08.349435 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:08.349416    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe\": container with ID starting with 5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe not found: ID does not exist" containerID="5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"
Oct 30 19:14:08.349508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.349448    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe"} err="failed to get container status \"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe\": rpc error: code = NotFound desc = could not find container \"5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe\": container with ID starting with 5e06e05f5c6f25643b7728d19a5023ade24dfb6630d85043386fefd25a821ebe not found: ID does not exist"
Oct 30 19:14:08.362013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.361959    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:14:08.369098 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:08.369062    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6c45zrvmm"]
Oct 30 19:14:09.952107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:09.947563    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53022857-f972-44e2-8aa4-4aae0700fc8e" path="/var/lib/kubelet/pods/53022857-f972-44e2-8aa4-4aae0700fc8e/volumes"
Oct 30 19:14:09.987468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:09.987433    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:14:09.988018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:09.987979    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy" containerID="cri-o://b1a73aca0de029ace8053a80991b4aaca48ef1f62765204aa63ee7c65fc6a92a" gracePeriod=30
Oct 30 19:14:10.276727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.276694    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:14:10.277625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277609    5065 cpu_manager.go:414] "containerMap: removing container" podUID="53022857-f972-44e2-8aa4-4aae0700fc8e" containerName="payload-processing"
Oct 30 19:14:10.277671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277627    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53022857-f972-44e2-8aa4-4aae0700fc8e" containerName="payload-processing"
Oct 30 19:14:10.277671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277645    5065 cpu_manager.go:414] "containerMap: removing container" podUID="97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" containerName="revoke-keys"
Oct 30 19:14:10.277671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277650    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" containerName="revoke-keys"
Oct 30 19:14:10.277793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277784    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="53022857-f972-44e2-8aa4-4aae0700fc8e" containerName="payload-processing"
Oct 30 19:14:10.277837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.277796    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" containerName="revoke-keys"
Oct 30 19:14:10.283580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.283561    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.304971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.304899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:14:10.335303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335268    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335397    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcfl\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335486    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335563    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335593    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335756    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335802    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.335922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.335860    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.336498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.336461    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.345213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.345190    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:14:10.351030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.351010    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.378615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.378420    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:14:10.438041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438007    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.438311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438206    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.438311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438248    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.438311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438288    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcfl\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.438495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438323    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrr9k\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.438495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438385    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.438495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438416    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.438495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438448    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.438751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438709    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.438822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.438779    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439020    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439140    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439207    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439274    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.439368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439308    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.439484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439443    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439497    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.439545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439529    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.439654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439562    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439829    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.439948 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439875    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.439948 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.439915    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.440231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.440211    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.442530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.442508    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.442621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.442506    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.458582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.458545    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.460469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.460447    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcfl\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\") pod \"maas-default-gateway-openshift-default-855f9574cf-s68fh\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.541972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.541905    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.541972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.541941    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542012    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542034    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542106    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542189    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542297    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrr9k\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542355    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542393    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542426    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542478    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542647    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.542712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.542689    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.543113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.543062    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.544761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.544739    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.544933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.544915    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.558957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.558926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.559190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.559167    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrr9k\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\") pod \"maas-default-gateway-openshift-default-855f9574cf-66lbb\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.603601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.603579    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:10.678846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.678615    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:10.913537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.913488    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:14:10.916243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.916204    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:10.916349 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.916314    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:10.916398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:10.916361    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:11.064586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.062025    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:14:11.071189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.071158    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:11.071302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.071224    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:11.071302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.071255    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:11.375037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.374991    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" event={"ID":"e497fb69-8848-428d-ade7-735218c8637c","Type":"ContainerStarted","Data":"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"}
Oct 30 19:14:11.375237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.375065    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" event={"ID":"e497fb69-8848-428d-ade7-735218c8637c","Type":"ContainerStarted","Data":"42e83501e8526c0fff446eeab47b33626e6be52d234f14efce0493fd2ef3f84e"}
Oct 30 19:14:11.377583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.377553    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" event={"ID":"8035a145-4645-400e-8237-79e4def5577e","Type":"ContainerStarted","Data":"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"}
Oct 30 19:14:11.377722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.377606    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" event={"ID":"8035a145-4645-400e-8237-79e4def5577e","Type":"ContainerStarted","Data":"0fc30b3125a4d1db5e70dce5cf080b8b3554ab925aca2fc9fab758d2791bb808"}
Oct 30 19:14:11.401225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.401179    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podStartSLOduration=1.4011619450000001 podStartE2EDuration="1.401161945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:11.400901135 +0000 UTC m=+2471.664366109" watchObservedRunningTime="2026-10-30 19:14:11.401161945 +0000 UTC m=+2471.664626892"
Oct 30 19:14:11.455166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.451920    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podStartSLOduration=1.451900833 podStartE2EDuration="1.451900833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:11.445169604 +0000 UTC m=+2471.708634565" watchObservedRunningTime="2026-10-30 19:14:11.451900833 +0000 UTC m=+2471.715365784"
Oct 30 19:14:11.609984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.609908    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:11.692578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.692189    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:11.791110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.770223    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:11.839569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.839538    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:11.847303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:11.847274    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.081157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.060173    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.088249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.088220    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.090029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.090002    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbzr\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.090885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.090862    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.193300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.193267    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbzr\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.193487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.193352    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.193769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.193748    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.193869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.193842    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.194122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.194063    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.194429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.194404    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.196659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.196634    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.212724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.212704    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbzr\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.425163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.416914    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:12.637472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.637420    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:12.651463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.650585    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:12.651463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.650641    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:12.666108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.658391    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:14:12.728249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.699221    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:12.728249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:12.699275    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:13.521305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.504282    5065 scope.go:122] "RemoveContainer" containerID="7cf4f90351f15063560e3b05bee073257ee21521deafe9a09267503529bb95c8"
Oct 30 19:14:13.924171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.898224    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:13.924171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.898290    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:13.968130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.968065    5065 scope.go:122] "RemoveContainer" containerID="2b71d2f2d75722264b7509e3bbf6e5a98a490ffe8302cdafb493bf68cc621c95"
Oct 30 19:14:13.974790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.974753    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:13.974918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:13.974810    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:14.062211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.061894    5065 generic.go:356] "Generic (PLEG): container finished" podID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerID="b44e4e6ec20240115d2254c9374d694bf6bfae83dce1dc7e0d5e3113ac8ef9ed" exitCode=0
Oct 30 19:14:14.067469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.067233    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerDied","Data":"b44e4e6ec20240115d2254c9374d694bf6bfae83dce1dc7e0d5e3113ac8ef9ed"}
Oct 30 19:14:14.067469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.067306    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:14.067469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.067346    5065 scope.go:122] "RemoveContainer" containerID="fdb6b654d4a5d264741fa9e0e275fb190f0b46c14349b012f969e28f53353664"
Oct 30 19:14:14.069847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.069656    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" event={"ID":"2f04bd30-e190-4332-95e1-453352574fce","Type":"ContainerStarted","Data":"33fe97b516644be67025a407a54ce36f8c26234b0471cdd298ba4ded84b2b91e"}
Oct 30 19:14:14.568500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.568475    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:14:14.617142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.616041    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:14.617142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.616126    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:14.675070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.675005    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.675396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.675350    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.680307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.680197    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.680632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.680570    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.687142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.680966    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.695539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.695472    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.701757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.701709    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.702170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.702055    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.702388 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.702349    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\") pod \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\" (UID: \"47fa508f-c8f1-49d2-a820-bf219fe97fb3\") "
Oct 30 19:14:14.712423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.712301    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.714504    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.714590    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.714634    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.715454    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.719167    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:14.726647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.720789    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:14.734894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.734842    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:14.853128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.852148    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.853128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.852182    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.853128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.852199    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.853128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.852214    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.865055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.862206    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "kube-api-access-wmtl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:14.865055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.864101    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:14.865055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.864484    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:14.867859 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.865973    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy" pod "47fa508f-c8f1-49d2-a820-bf219fe97fb3" (UID: "47fa508f-c8f1-49d2-a820-bf219fe97fb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:14.963104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.962773    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.963104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.962806    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.963104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.962819    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47fa508f-c8f1-49d2-a820-bf219fe97fb3-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:14.963104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:14.962831    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmtl4\" (UniqueName: \"kubernetes.io/projected/47fa508f-c8f1-49d2-a820-bf219fe97fb3-kube-api-access-wmtl4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.149178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.149144    5065 generic.go:356] "Generic (PLEG): container finished" podID="3d52305f-1374-4f58-b021-693c818391bd" containerID="b1a73aca0de029ace8053a80991b4aaca48ef1f62765204aa63ee7c65fc6a92a" exitCode=0
Oct 30 19:14:15.149455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.149210    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerDied","Data":"b1a73aca0de029ace8053a80991b4aaca48ef1f62765204aa63ee7c65fc6a92a"}
Oct 30 19:14:15.149455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.149308    5065 scope.go:122] "RemoveContainer" containerID="082c5628cc7d227e9422e9575248af0d2d88768c27467a923d3336c4d794410f"
Oct 30 19:14:15.152970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.152938    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" event={"ID":"2f04bd30-e190-4332-95e1-453352574fce","Type":"ContainerStarted","Data":"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"}
Oct 30 19:14:15.153106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.153031    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:15.154870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.154849    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"
Oct 30 19:14:15.154988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.154950    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb" event={"ID":"47fa508f-c8f1-49d2-a820-bf219fe97fb3","Type":"ContainerDied","Data":"8bcc486c3f3dfbb94efcff7268a95b98ede8c6f84b67a98f1adf17816cc73465"}
Oct 30 19:14:15.171825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.171806    5065 scope.go:122] "RemoveContainer" containerID="b44e4e6ec20240115d2254c9374d694bf6bfae83dce1dc7e0d5e3113ac8ef9ed"
Oct 30 19:14:15.241162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.213704    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" podStartSLOduration=4.213684757 podStartE2EDuration="4.213684757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:15.176742392 +0000 UTC m=+2475.440207360" watchObservedRunningTime="2026-10-30 19:14:15.213684757 +0000 UTC m=+2475.477149706"
Oct 30 19:14:15.264543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.264477    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 19:14:15.271437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.271411    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-whssb"]
Oct 30 19:14:15.609031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.608983    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:15.609669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.609622    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:15.675206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.675179    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:14:15.700401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.700240    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:15.700401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.700288    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:15.718874 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.718848    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719120    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719264    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719373    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719492    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719696    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.719916 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719902    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.720172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.720152    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.720329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.720313    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\") pod \"3d52305f-1374-4f58-b021-693c818391bd\" (UID: \"3d52305f-1374-4f58-b021-693c818391bd\") "
Oct 30 19:14:15.725115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.719737    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.725115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.723062    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.725467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.725446    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:15.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.732159    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.737132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.732228    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.746464 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.746437    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:15.749111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.748852    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:15.749111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.748958    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:15.749920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.749885    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l" pod "3d52305f-1374-4f58-b021-693c818391bd" (UID: "3d52305f-1374-4f58-b021-693c818391bd"). InnerVolumeSpecName "kube-api-access-2pd7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849200    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pd7l\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-kube-api-access-2pd7l\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849241    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d52305f-1374-4f58-b021-693c818391bd-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849257    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849272    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d52305f-1374-4f58-b021-693c818391bd-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849291    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d52305f-1374-4f58-b021-693c818391bd-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849306    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849320    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849336    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.849392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.849362    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d52305f-1374-4f58-b021-693c818391bd-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.976521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:15.976429    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" path="/var/lib/kubelet/pods/47fa508f-c8f1-49d2-a820-bf219fe97fb3/volumes"
Oct 30 19:14:16.166129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.166069    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"
Oct 30 19:14:16.166309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.166132    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g" event={"ID":"3d52305f-1374-4f58-b021-693c818391bd","Type":"ContainerDied","Data":"c1d64568d9f6a1066f36d0cccd92144f8543c88cba89ec16489f6b04781611e5"}
Oct 30 19:14:16.166309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.166197    5065 scope.go:122] "RemoveContainer" containerID="b1a73aca0de029ace8053a80991b4aaca48ef1f62765204aa63ee7c65fc6a92a"
Oct 30 19:14:16.242174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.241630    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:14:16.253627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.253595    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9v79g"]
Oct 30 19:14:16.604150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.604112    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:16.610116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.604424    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:16.684886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.684847    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:16.685944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:16.685908    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:17.608013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.607977    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:17.608328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.608295    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:17.659362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.659156    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" podUID="9b273358-1820-4ab4-b065-5b07aab2b98b" containerName="payload-processing" containerID="cri-o://5eb287eebee2971f43e4a83010efadbdd34714fea8149657cf9d26d4b4ad2a77" gracePeriod=25
Oct 30 19:14:17.686275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.679406    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:17.686275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.679463    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:17.767974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.767231    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:14:17.767974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:17.767440    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy" containerID="cri-o://99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66" gracePeriod=30
Oct 30 19:14:18.528175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.505376    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d52305f-1374-4f58-b021-693c818391bd" path="/var/lib/kubelet/pods/3d52305f-1374-4f58-b021-693c818391bd/volumes"
Oct 30 19:14:18.670629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.663231    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:18.670629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.663290    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:18.705347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.683786    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:18.705347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.683839    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:18.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.719722    5065 generic.go:356] "Generic (PLEG): container finished" podID="9b273358-1820-4ab4-b065-5b07aab2b98b" containerID="5eb287eebee2971f43e4a83010efadbdd34714fea8149657cf9d26d4b4ad2a77" exitCode=0
Oct 30 19:14:18.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:18.720022    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" event={"ID":"9b273358-1820-4ab4-b065-5b07aab2b98b","Type":"ContainerDied","Data":"5eb287eebee2971f43e4a83010efadbdd34714fea8149657cf9d26d4b4ad2a77"}
Oct 30 19:14:19.502131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.500351    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:14:19.555137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.555078    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:19.556580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556549    5065 cpu_manager.go:414] "containerMap: removing container" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:19.556580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556577    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556596    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556607    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556664    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9b273358-1820-4ab4-b065-5b07aab2b98b" containerName="payload-processing"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556673    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b273358-1820-4ab4-b065-5b07aab2b98b" containerName="payload-processing"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556705    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556713    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556746    5065 cpu_manager.go:414] "containerMap: removing container" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:19.556768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556753    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:19.557235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.556981    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b273358-1820-4ab4-b065-5b07aab2b98b" containerName="payload-processing"
Oct 30 19:14:19.557235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.557002    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:19.557235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.557016    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:19.564556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.563698    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:19.564556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.563805    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.572104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.571420    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4040011dcd71\""
Oct 30 19:14:19.574783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.572979    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\") pod \"9b273358-1820-4ab4-b065-5b07aab2b98b\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") "
Oct 30 19:14:19.574783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.573054    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\") pod \"9b273358-1820-4ab4-b065-5b07aab2b98b\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") "
Oct 30 19:14:19.574783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.573159    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\") pod \"9b273358-1820-4ab4-b065-5b07aab2b98b\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") "
Oct 30 19:14:19.574783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.573413    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\") pod \"9b273358-1820-4ab4-b065-5b07aab2b98b\" (UID: \"9b273358-1820-4ab4-b065-5b07aab2b98b\") "
Oct 30 19:14:19.575073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.574961    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume" pod "9b273358-1820-4ab4-b065-5b07aab2b98b" (UID: "9b273358-1820-4ab4-b065-5b07aab2b98b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:19.578192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.575230    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume" pod "9b273358-1820-4ab4-b065-5b07aab2b98b" (UID: "9b273358-1820-4ab4-b065-5b07aab2b98b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:19.590835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.578937    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials" pod "9b273358-1820-4ab4-b065-5b07aab2b98b" (UID: "9b273358-1820-4ab4-b065-5b07aab2b98b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:19.614345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.604927    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:19.614345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.604976    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:19.649309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.649271    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg" pod "9b273358-1820-4ab4-b065-5b07aab2b98b" (UID: "9b273358-1820-4ab4-b065-5b07aab2b98b"). InnerVolumeSpecName "kube-api-access-hnwrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:19.676395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.676369    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.676721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.676528    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pb6d\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.676721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.676578    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.676721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.676651    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.677050 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.677029    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.677111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.677054    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwrg\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-kube-api-access-hnwrg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.677111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.677067    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b273358-1820-4ab4-b065-5b07aab2b98b-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.677111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.677081    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b273358-1820-4ab4-b065-5b07aab2b98b-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.700106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.699552    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:19.700106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.699619    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:19.753702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.753666    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk" event={"ID":"9b273358-1820-4ab4-b065-5b07aab2b98b","Type":"ContainerDied","Data":"fee458c4e948b8358964d6ec54e0f8ede15f4a64ce7d27e753bd9176199f8145"}
Oct 30 19:14:19.753915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.753743    5065 scope.go:122] "RemoveContainer" containerID="5eb287eebee2971f43e4a83010efadbdd34714fea8149657cf9d26d4b4ad2a77"
Oct 30 19:14:19.753915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.753867    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"
Oct 30 19:14:19.778627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.778596    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.778949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.778928    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.779038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.778996    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb6d\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.779126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.779050    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.791770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.791748    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.792927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.792617    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.821107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.821039    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pb6d\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.821263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.821121    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:14:19.822133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.822109    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\") pod \"payload-processing-external-model-775444565c-7s4lp\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") " pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:19.835821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.835798    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6d8fz24sk"]
Oct 30 19:14:19.940908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.940820    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b273358-1820-4ab4-b065-5b07aab2b98b" path="/var/lib/kubelet/pods/9b273358-1820-4ab4-b065-5b07aab2b98b/volumes"
Oct 30 19:14:19.954857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:19.954835    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:20.154610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.153772    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" podUID="2f04bd30-e190-4332-95e1-453352574fce" containerName="payload-processing" containerID="cri-o://f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4" gracePeriod=25
Oct 30 19:14:20.359593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.359556    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:20.365019 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:14:20.364964    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b597c8_2ac7_45c3_85c4_aa645b7fabba.slice/crio-08eb94807d028631e3e0870c8924afe74b558a76628020ec5707775acd127b66 WatchSource:0}: Error finding container 08eb94807d028631e3e0870c8924afe74b558a76628020ec5707775acd127b66: Status 404 returned error can't find the container with id 08eb94807d028631e3e0870c8924afe74b558a76628020ec5707775acd127b66
Oct 30 19:14:20.606721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.605411    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:20.606721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.605466    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:20.680425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.679854    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:20.680425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.679915    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:20.723762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.723716    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:20.771315 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.770768    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" event={"ID":"19b597c8-2ac7-45c3-85c4-aa645b7fabba","Type":"ContainerStarted","Data":"301e03a5ce79649ce8e7712feca3faec343f080a8a7c35e17cf5e36edb9f0815"}
Oct 30 19:14:20.771315 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.770820    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" event={"ID":"19b597c8-2ac7-45c3-85c4-aa645b7fabba","Type":"ContainerStarted","Data":"08eb94807d028631e3e0870c8924afe74b558a76628020ec5707775acd127b66"}
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.771727    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.773018    5065 generic.go:356] "Generic (PLEG): container finished" podID="2f04bd30-e190-4332-95e1-453352574fce" containerID="f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4" exitCode=0
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.773056    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.773224    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" event={"ID":"2f04bd30-e190-4332-95e1-453352574fce","Type":"ContainerDied","Data":"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"}
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.773331    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf" event={"ID":"2f04bd30-e190-4332-95e1-453352574fce","Type":"ContainerDied","Data":"33fe97b516644be67025a407a54ce36f8c26234b0471cdd298ba4ded84b2b91e"}
Oct 30 19:14:20.773990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.773368    5065 scope.go:122] "RemoveContainer" containerID="f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"
Oct 30 19:14:20.794466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.794443    5065 scope.go:122] "RemoveContainer" containerID="f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"
Oct 30 19:14:20.794966 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:20.794941    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4\": container with ID starting with f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4 not found: ID does not exist" containerID="f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"
Oct 30 19:14:20.795137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.795111    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4"} err="failed to get container status \"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4\": rpc error: code = NotFound desc = could not find container \"f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4\": container with ID starting with f362c435c6b56aa20a5e9be6bcc8d9ec8511566b580d08280c9ac9e817edcce4 not found: ID does not exist"
Oct 30 19:14:20.850824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.850777    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" podStartSLOduration=1.8507643539999998 podStartE2EDuration="1.850764354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:20.847232962 +0000 UTC m=+2481.110697952" watchObservedRunningTime="2026-10-30 19:14:20.850764354 +0000 UTC m=+2481.114229301"
Oct 30 19:14:20.866286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.866215    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\") pod \"2f04bd30-e190-4332-95e1-453352574fce\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") "
Oct 30 19:14:20.866410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.866358    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\") pod \"2f04bd30-e190-4332-95e1-453352574fce\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") "
Oct 30 19:14:20.866491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.866416    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\") pod \"2f04bd30-e190-4332-95e1-453352574fce\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") "
Oct 30 19:14:20.866654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.866637    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\") pod \"2f04bd30-e190-4332-95e1-453352574fce\" (UID: \"2f04bd30-e190-4332-95e1-453352574fce\") "
Oct 30 19:14:20.867483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.866817    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume" pod "2f04bd30-e190-4332-95e1-453352574fce" (UID: "2f04bd30-e190-4332-95e1-453352574fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:20.867483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.867444    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume" pod "2f04bd30-e190-4332-95e1-453352574fce" (UID: "2f04bd30-e190-4332-95e1-453352574fce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:20.868451 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.867838    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:20.868451 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.867864    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f04bd30-e190-4332-95e1-453352574fce-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:20.870284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.870226    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials" pod "2f04bd30-e190-4332-95e1-453352574fce" (UID: "2f04bd30-e190-4332-95e1-453352574fce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:20.870909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.870880    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr" pod "2f04bd30-e190-4332-95e1-453352574fce" (UID: "2f04bd30-e190-4332-95e1-453352574fce"). InnerVolumeSpecName "kube-api-access-wgbzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:20.969592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.969556    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:20.969592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:20.969590    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgbzr\" (UniqueName: \"kubernetes.io/projected/2f04bd30-e190-4332-95e1-453352574fce-kube-api-access-wgbzr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:21.108902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.108576    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:21.118654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.118571    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-f97dmnxdf"]
Oct 30 19:14:21.604743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.604704    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-s68fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body=
Oct 30 19:14:21.604908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.604773    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused"
Oct 30 19:14:21.681171 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.681127    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-66lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body=
Oct 30 19:14:21.681642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.681186    5065 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused"
Oct 30 19:14:21.901159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:21.898740    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f04bd30-e190-4332-95e1-453352574fce" path="/var/lib/kubelet/pods/2f04bd30-e190-4332-95e1-453352574fce/volumes"
Oct 30 19:14:22.875248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:22.875208    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:22.879130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:22.876260    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:22.881015 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:22.880503    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:14:23.148935 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:23.148828    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:14:23.289650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.289613    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:14:23.384510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.384420    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.384971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.384947    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.385286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385264    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.385429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385415    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.385567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385553    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.385789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385773    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.385957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385942    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.386292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.386271    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.386478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.386464    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\") pod \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\" (UID: \"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a\") "
Oct 30 19:14:23.388407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.384971    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:23.388547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.385054    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:23.388648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.386599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:23.388743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.386932    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:23.388846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.387607    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:23.389313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.389293    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.389433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.389420    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.389539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.389527    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.389624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.389613    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.389718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.389704    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.399545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.399499    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:23.406112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.406048    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:23.406569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.406525    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:23.407811 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.407666    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2" pod "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" (UID: "27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a"). InnerVolumeSpecName "kube-api-access-2rdw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:23.506041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.499425    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.507291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.506631    5065 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:23.509443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.509375    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:23.509597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.509578    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.510456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.510424    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdw2\" (UniqueName: \"kubernetes.io/projected/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-kube-api-access-2rdw2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.510592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.510579    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:23.529633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.529540    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:14:23.838039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.838006    5065 generic.go:356] "Generic (PLEG): container finished" podID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerID="99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66" exitCode=0
Oct 30 19:14:23.838345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.838057    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"
Oct 30 19:14:23.838559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.838075    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" event={"ID":"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a","Type":"ContainerDied","Data":"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"}
Oct 30 19:14:23.839730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.839697    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7" event={"ID":"27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a","Type":"ContainerDied","Data":"f87fbeafe04ed3b09601763303287cb91c18da296680bfea7e7366642c571937"}
Oct 30 19:14:23.839871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.839753    5065 scope.go:122] "RemoveContainer" containerID="99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"
Oct 30 19:14:23.867465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.867447    5065 scope.go:122] "RemoveContainer" containerID="99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"
Oct 30 19:14:23.867840 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:23.867819    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66\": container with ID starting with 99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66 not found: ID does not exist" containerID="99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"
Oct 30 19:14:23.867908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.867848    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66"} err="failed to get container status \"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66\": rpc error: code = NotFound desc = could not find container \"99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66\": container with ID starting with 99b28b56bd95b2764f86e4c857d2d60af87a48cd54c62a17ac943c56b0c57d66 not found: ID does not exist"
Oct 30 19:14:23.902912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.902881    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:14:23.910905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:23.910873    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-srvs7"]
Oct 30 19:14:24.480140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:24.472706    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:26.031548 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:26.031509    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" path="/var/lib/kubelet/pods/27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a/volumes"
Oct 30 19:14:29.511178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:29.493119    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" podUID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" containerName="payload-processing" containerID="cri-o://301e03a5ce79649ce8e7712feca3faec343f080a8a7c35e17cf5e36edb9f0815" gracePeriod=25
Oct 30 19:14:29.562261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:29.562186    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:29.608332 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:29.608269    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:14:29.672140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:29.671381    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 19:14:29.699166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:29.696516    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:30.020587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.019410    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:30.034971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.021460    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:30.439170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.434648    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:30.576221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.576119    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 19:14:30.577442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.577404    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:14:30.756077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.752646    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:14:30.756077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.754896    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:14:30.812035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.794505    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:14:30.812035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:30.794611    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:14:31.090959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.090913    5065 generic.go:356] "Generic (PLEG): container finished" podID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" containerID="301e03a5ce79649ce8e7712feca3faec343f080a8a7c35e17cf5e36edb9f0815" exitCode=0
Oct 30 19:14:31.091790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.091708    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" event={"ID":"19b597c8-2ac7-45c3-85c4-aa645b7fabba","Type":"ContainerDied","Data":"301e03a5ce79649ce8e7712feca3faec343f080a8a7c35e17cf5e36edb9f0815"}
Oct 30 19:14:31.137126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.134704    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:31.137126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.135793    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:14:31.327137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.312552    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:14:31.329284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.328663    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:31.330210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.330070    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:14:31.422212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.412445    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\") pod \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") "
Oct 30 19:14:31.422212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.412691    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\") pod \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") "
Oct 30 19:14:31.422212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.412749    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\") pod \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") "
Oct 30 19:14:31.530407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.511035    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume" pod "19b597c8-2ac7-45c3-85c4-aa645b7fabba" (UID: "19b597c8-2ac7-45c3-85c4-aa645b7fabba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:31.530919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.530885    5065 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-p8smm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:14:31.621435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.612434    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-p8smm" podUID="ea311f74-4221-4d7a-a3da-bc4a30848044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:31.745110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.733677    5065 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:31.745110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.734700    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:31.745110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.738273    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials" pod "19b597c8-2ac7-45c3-85c4-aa645b7fabba" (UID: "19b597c8-2ac7-45c3-85c4-aa645b7fabba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:31.745110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.739202    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 30 19:14:31.957110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.831765    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\") pod \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\" (UID: \"19b597c8-2ac7-45c3-85c4-aa645b7fabba\") "
Oct 30 19:14:31.957110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.881429    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume" pod "19b597c8-2ac7-45c3-85c4-aa645b7fabba" (UID: "19b597c8-2ac7-45c3-85c4-aa645b7fabba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:31.957110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.946699    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.957110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.946783    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.957110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.946834    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b597c8-2ac7-45c3-85c4-aa645b7fabba-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.963936 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.963892    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:14:31.964264 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.964236    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 30 19:14:32.041247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.992803    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:14:32.331765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:31.993756    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:14:32.419128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.402217    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:32.419128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.183148    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:32.535664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.047692    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d" pod "19b597c8-2ac7-45c3-85c4-aa645b7fabba" (UID: "19b597c8-2ac7-45c3-85c4-aa645b7fabba"). InnerVolumeSpecName "kube-api-access-7pb6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:32.594160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.593683    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pb6d\" (UniqueName: \"kubernetes.io/projected/19b597c8-2ac7-45c3-85c4-aa645b7fabba-kube-api-access-7pb6d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:32.615951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.615458    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp"
Oct 30 19:14:32.628245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.628208    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-775444565c-7s4lp" event={"ID":"19b597c8-2ac7-45c3-85c4-aa645b7fabba","Type":"ContainerDied","Data":"08eb94807d028631e3e0870c8924afe74b558a76628020ec5707775acd127b66"}
Oct 30 19:14:32.628728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.628371    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:14:32.628728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.628418    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:14:32.628990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.628973    5065 scope.go:122] "RemoveContainer" containerID="301e03a5ce79649ce8e7712feca3faec343f080a8a7c35e17cf5e36edb9f0815"
Oct 30 19:14:32.737546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.737492    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:32.857609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.855618    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7"
Oct 30 19:14:32.880776 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.880746    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:14:32.908890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.896237    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:32.908890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.899713    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:14:32.932108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.918185    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:14:32.950371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:32.950282    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-775444565c-7s4lp"]
Oct 30 19:14:32.953179 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:32.953150    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b597c8_2ac7_45c3_85c4_aa645b7fabba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.017451    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018797    5065 cpu_manager.go:414] "containerMap: removing container" podUID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" containerName="payload-processing"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018817    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" containerName="payload-processing"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018867    5065 cpu_manager.go:414] "containerMap: removing container" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018875    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018886    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2f04bd30-e190-4332-95e1-453352574fce" containerName="payload-processing"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.018893    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f04bd30-e190-4332-95e1-453352574fce" containerName="payload-processing"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.019080    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e68cd6-acb9-4c00-b3bd-4bf1089cbb2a" containerName="istio-proxy"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.019119    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f04bd30-e190-4332-95e1-453352574fce" containerName="payload-processing"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.019138    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d52305f-1374-4f58-b021-693c818391bd" containerName="istio-proxy"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.019153    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fa508f-c8f1-49d2-a820-bf219fe97fb3" containerName="istio-proxy"
Oct 30 19:14:33.019716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.019163    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" containerName="payload-processing"
Oct 30 19:14:33.028303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.025812    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.032947    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-e2e--a67c8769-dockercfg-czgjj\""
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.033227    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"kube-root-ca.crt\""
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.033299    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"openshift-service-ca.crt\""
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.033383    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c6027\""
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.033444    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"routing-overlay\""
Oct 30 19:14:33.037920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.033496    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-b77e998367e9\""
Oct 30 19:14:33.053889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.053858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:14:33.131929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.131833    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9ds\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.132281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.132257    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.136172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.135205    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.136681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.136653    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.249612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.249572    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9ds\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.249905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.249881    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.250215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.250192    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.250604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.250583    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.272109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.251626    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.272318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.260978    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.305834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.305800    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9ds\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.308306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.308188    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:33.363660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:33.363367    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:34.371722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.370191    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b597c8-2ac7-45c3-85c4-aa645b7fabba" path="/var/lib/kubelet/pods/19b597c8-2ac7-45c3-85c4-aa645b7fabba/volumes"
Oct 30 19:14:34.371722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.370845    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:14:34.383138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.382432    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:14:34.383138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.382552    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.388240 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.388211    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-cb587320c193\""
Oct 30 19:14:34.388493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.388472    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-e2e--0575c797-dockercfg-2h4h6\""
Oct 30 19:14:34.390611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.390595    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"routing-overlay\""
Oct 30 19:14:34.390791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.390779    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-plugins-e2e-extmod-a-de4bda\""
Oct 30 19:14:34.390948 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.390937    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"kube-root-ca.crt\""
Oct 30 19:14:34.391492 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.391477    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"openshift-service-ca.crt\""
Oct 30 19:14:34.463063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.463034    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnn9p\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.463260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.463105    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.463314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.463262    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.463951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.463524    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.473684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.473662    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:14:34.476318 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:14:34.476296    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8ab131_db2a_441c_a81f_9e283ef92bbe.slice/crio-593ca28f7fe2b5ae3863118ad1772f09a15a5b639eb2ae8f98b6e9d95f6688a6 WatchSource:0}: Error finding container 593ca28f7fe2b5ae3863118ad1772f09a15a5b639eb2ae8f98b6e9d95f6688a6: Status 404 returned error can't find the container with id 593ca28f7fe2b5ae3863118ad1772f09a15a5b639eb2ae8f98b6e9d95f6688a6
Oct 30 19:14:34.565205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.565172    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.565525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.565505    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.565643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.565624    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnn9p\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.565716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.565691    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.565796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.565779    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.566555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.566532    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.568489 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.568464    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.579426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.579402    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnn9p\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.713234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.711732    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:34.739167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.739127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" event={"ID":"6d8ab131-db2a-441c-a81f-9e283ef92bbe","Type":"ContainerStarted","Data":"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"}
Oct 30 19:14:34.739334 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.739201    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" event={"ID":"6d8ab131-db2a-441c-a81f-9e283ef92bbe","Type":"ContainerStarted","Data":"593ca28f7fe2b5ae3863118ad1772f09a15a5b639eb2ae8f98b6e9d95f6688a6"}
Oct 30 19:14:34.740616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.740588    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:34.780359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:34.780311    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" podStartSLOduration=2.780297205 podStartE2EDuration="2.780297205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:34.775552815 +0000 UTC m=+2495.039017782" watchObservedRunningTime="2026-10-30 19:14:34.780297205 +0000 UTC m=+2495.043762152"
Oct 30 19:14:35.047857 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:14:35.047819    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0af92be_b39b_4a2a_b760_73e4986ce05a.slice/crio-b4f7c797fa1819ff953fcd0f065459f8ad0eb0c2405cfa79b5d1a45345a70d64 WatchSource:0}: Error finding container b4f7c797fa1819ff953fcd0f065459f8ad0eb0c2405cfa79b5d1a45345a70d64: Status 404 returned error can't find the container with id b4f7c797fa1819ff953fcd0f065459f8ad0eb0c2405cfa79b5d1a45345a70d64
Oct 30 19:14:35.051995 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:35.051966    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:14:35.784119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:35.782654    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" event={"ID":"a0af92be-b39b-4a2a-b760-73e4986ce05a","Type":"ContainerStarted","Data":"f296efdc520b4b92dbef0a1eecb2d99c9b73227b7c166a7766c34d98d1770630"}
Oct 30 19:14:35.784119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:35.782714    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" event={"ID":"a0af92be-b39b-4a2a-b760-73e4986ce05a","Type":"ContainerStarted","Data":"b4f7c797fa1819ff953fcd0f065459f8ad0eb0c2405cfa79b5d1a45345a70d64"}
Oct 30 19:14:35.784119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:35.782758    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:35.821725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:35.821676    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" podStartSLOduration=2.821661264 podStartE2EDuration="2.821661264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:35.817573636 +0000 UTC m=+2496.081038583" watchObservedRunningTime="2026-10-30 19:14:35.821661264 +0000 UTC m=+2496.085126212"
Oct 30 19:14:43.020614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:43.020555    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:14:43.021227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:43.020829    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy" containerID="cri-o://12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0" gracePeriod=30
Oct 30 19:14:46.831376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:46.831344    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:14:47.839153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:47.839079    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:14:48.365067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.365043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:14:48.389115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389038    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389096    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389129    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389250    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389333    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389336    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:48.389610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389459    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389610 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389571    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389638    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389720    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\") pod \"64930f85-2d31-4da0-8efc-53ab7f4f3721\" (UID: \"64930f85-2d31-4da0-8efc-53ab7f4f3721\") "
Oct 30 19:14:48.389852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.389782    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390172    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390180    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390581    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390603    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390618    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64930f85-2d31-4da0-8efc-53ab7f4f3721-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390632    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.390769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.390693    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:48.393076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.392528    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:48.393530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.393490    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:48.394603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.394565    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "kube-api-access-qv78g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:48.399822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.395556    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy" pod "64930f85-2d31-4da0-8efc-53ab7f4f3721" (UID: "64930f85-2d31-4da0-8efc-53ab7f4f3721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:48.492203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.492171    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv78g\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-kube-api-access-qv78g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.492203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.492199    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.492203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.492210    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.492502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.492219    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:48.492502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:48.492227    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64930f85-2d31-4da0-8efc-53ab7f4f3721-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.080768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.080730    5065 generic.go:356] "Generic (PLEG): container finished" podID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerID="12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0" exitCode=0
Oct 30 19:14:49.081584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.080778    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerDied","Data":"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"}
Oct 30 19:14:49.081584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.080813    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"
Oct 30 19:14:49.081584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.080835    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr" event={"ID":"64930f85-2d31-4da0-8efc-53ab7f4f3721","Type":"ContainerDied","Data":"b513324b85300cfb6e0f0c916982492f46de02532458563ce9f7dd6c016b8a61"}
Oct 30 19:14:49.081584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.080858    5065 scope.go:122] "RemoveContainer" containerID="12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"
Oct 30 19:14:49.095642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.095608    5065 scope.go:122] "RemoveContainer" containerID="43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"
Oct 30 19:14:49.111689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.111668    5065 scope.go:122] "RemoveContainer" containerID="12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"
Oct 30 19:14:49.112555 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:49.112528    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0\": container with ID starting with 12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0 not found: ID does not exist" containerID="12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"
Oct 30 19:14:49.112690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.112566    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0"} err="failed to get container status \"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0\": rpc error: code = NotFound desc = could not find container \"12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0\": container with ID starting with 12f87060015507e0b75723bfd182035b9335bf89f5a5f7610a60a59ac3e0a0e0 not found: ID does not exist"
Oct 30 19:14:49.112690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.112586    5065 scope.go:122] "RemoveContainer" containerID="43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"
Oct 30 19:14:49.113080 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:49.113053    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87\": container with ID starting with 43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87 not found: ID does not exist" containerID="43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"
Oct 30 19:14:49.113560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.113530    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87"} err="failed to get container status \"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87\": rpc error: code = NotFound desc = could not find container \"43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87\": container with ID starting with 43444a2af7b110847b6f4430e1b65c1db49eccc1683d9c4f1211f953a2ea3e87 not found: ID does not exist"
Oct 30 19:14:49.114008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.113992    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:14:49.119710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.119687    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-np6hr"]
Oct 30 19:14:49.575182 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.575151    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:14:49.576110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.576073    5065 cpu_manager.go:414] "containerMap: removing container" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:14:49.576110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.576111    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:14:49.576302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.576148    5065 cpu_manager.go:414] "containerMap: removing container" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:14:49.576302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.576173    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:14:49.576460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.576445    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:14:49.582002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.581981    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.598812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.598688    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:14:49.622440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.622407    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.622770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.622745    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhpg\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.622974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.622955    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.623192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.623171    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.623491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.623470    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.623648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.623631    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.623854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.623837    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.725984    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726069    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhpg\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726125    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726191    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726293    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726334    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.726483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.726391    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.727351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.727325    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.729011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.728983    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.737148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.729889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.737148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.730137    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.737148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.730291    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.741379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.741353    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.742135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.742056    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhpg\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jm7mp\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:49.902225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.902120    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" path="/var/lib/kubelet/pods/64930f85-2d31-4da0-8efc-53ab7f4f3721/volumes"
Oct 30 19:14:49.909372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:49.909342    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:50.239419 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:14:50.239383    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbec99a3_129b_4049_92ff_ac6f73c69e54.slice/crio-31cb6bfd93fb486d8776f852e6eb69693b3206dc5cbdf5287444bad8f9ec549f WatchSource:0}: Error finding container 31cb6bfd93fb486d8776f852e6eb69693b3206dc5cbdf5287444bad8f9ec549f: Status 404 returned error can't find the container with id 31cb6bfd93fb486d8776f852e6eb69693b3206dc5cbdf5287444bad8f9ec549f
Oct 30 19:14:50.240854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:50.240827    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:14:50.243059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:50.243018    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:50.243200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:50.243117    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:14:51.152202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:51.152167    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" event={"ID":"fbec99a3-129b-4049-92ff-ac6f73c69e54","Type":"ContainerStarted","Data":"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"}
Oct 30 19:14:51.152498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:51.152469    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" event={"ID":"fbec99a3-129b-4049-92ff-ac6f73c69e54","Type":"ContainerStarted","Data":"31cb6bfd93fb486d8776f852e6eb69693b3206dc5cbdf5287444bad8f9ec549f"}
Oct 30 19:14:51.154311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:51.153748    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:51.167153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:51.165860    5065 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jm7mp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:14:51.167153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:51.165914    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:52.162747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:52.162710    5065 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jm7mp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:14:52.163284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:52.162765    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:53.060340 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:14:53.060300    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:14:53.173338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:53.173311    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:14:53.208821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:14:53.208762    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" podStartSLOduration=4.208743878 podStartE2EDuration="4.208743878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:51.231604139 +0000 UTC m=+2511.495069089" watchObservedRunningTime="2026-10-30 19:14:53.208743878 +0000 UTC m=+2513.472208827"
Oct 30 19:15:00.168720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.168687    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:00.169759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.169741    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="64930f85-2d31-4da0-8efc-53ab7f4f3721" containerName="istio-proxy"
Oct 30 19:15:00.173919 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.173898    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.188259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.188234    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:00.188259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.188255    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:15:00.195524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.195501    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.197791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.197767    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:00.217153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.205265    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.217153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.207721    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:00.230718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.230561    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.230888 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.230870    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:00.291734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.291667    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp595\" (UniqueName: \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.303412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.303366    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.304295 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.304270    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb28s\" (UniqueName: \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.304633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.304606    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2rx\" (UniqueName: \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.305145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.305122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.305455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.305426    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw8n\" (UniqueName: \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.305699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.305677    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.305903 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.305883    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.410107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.409187    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2rx\" (UniqueName: \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.410107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.409466    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.410107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.409775    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw8n\" (UniqueName: \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.410733    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.410760    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.411269    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.412164    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.423188    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.423423    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp595\" (UniqueName: \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.423514    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.436160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.423630    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb28s\" (UniqueName: \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.452214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.450574    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.468164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.461862    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:00.468164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.461920    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:00.468164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.461947    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:00.468164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.461973    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:00.468164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.463332    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.469856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.469833    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:00.470233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.470214    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.473988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.473951    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:00.508109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.503254    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.515484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.509079    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw8n\" (UniqueName: \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\") pod \"maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.515484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.509155    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb28s\" (UniqueName: \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\") pod \"maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.517198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.516916    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2rx\" (UniqueName: \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.525914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.525858    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp595\" (UniqueName: \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.527251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527216    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.527410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527392    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.527459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527433    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.527516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527501    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lfn\" (UniqueName: \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.527573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527558    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.527688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.527670    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6c5k\" (UniqueName: \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.534290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.534247    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:00.543440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.543404    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.544429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.544395    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:00.556152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.556124    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.557387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.557363    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:00.563257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.563230    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.567801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.567773    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:00.584971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.579060    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:00.611161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.611077    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:00.612753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.612721    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:15:00.618931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.618647    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:00.627049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.625847    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:00.629808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.629771    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2d5z\" (UniqueName: \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.630444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.630390    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.631046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.630687    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.631306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.631286    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lfn\" (UniqueName: \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.631502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.631466    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.631729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.631708    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc2lf\" (UniqueName: \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.631953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.631936    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6c5k\" (UniqueName: \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.632169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.632079    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.632407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.632388    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.632630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.632613    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.633133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.633111    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.633529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.633510    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspl6\" (UniqueName: \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.633778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.633753    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.633867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.631338    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.633932 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.632644    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.634208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.634185    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.642841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.641181    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:00.648286 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.648259    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:00.648480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.648380    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.654691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.654657    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.654933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.654817    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lfn\" (UniqueName: \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\") pod \"maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.655893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.655867    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.656029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.655898    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6c5k\" (UniqueName: \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\") pod \"maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.662232 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.657591    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:00.664638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.664606    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.682531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.682500    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:00.683387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.683364    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:00.699606 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.699570    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:00.705406 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.705299    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:00.729906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.723607    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.740936    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741140    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspl6\" (UniqueName: \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741190    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741230    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741257    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741296    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2d5z\" (UniqueName: \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741438    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bll8\" (UniqueName: \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741518    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6cm\" (UniqueName: \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741584    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741613    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc2lf\" (UniqueName: \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741677    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741745    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttzr\" (UniqueName: \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741782    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.742127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.741806    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55ft\" (UniqueName: \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.745808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.743122    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:00.745808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.743597    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.745808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.744882    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.745808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.745101    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.748111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.748035    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.752342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.752120    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:00.752342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.752242    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.766022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.765993    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:00.771328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.770674    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2d5z\" (UniqueName: \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\") pod \"maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.771328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.771198    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:00.772498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.772473    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspl6\" (UniqueName: \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.773473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.773441    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc2lf\" (UniqueName: \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.784980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.782416    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:00.814564 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.814531    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:00.844422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.844382    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.852207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.852174    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.852568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.852546    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.852849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.852821    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.853040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.853025    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bll8\" (UniqueName: \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.853250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.853230    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6cm\" (UniqueName: \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.853775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.853755    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.855946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.855916    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5hr\" (UniqueName: \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.856255 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.856231    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttzr\" (UniqueName: \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.856493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.856472    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55ft\" (UniqueName: \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.856782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.856763    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.857198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.857176    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgcv\" (UniqueName: \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.857599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.854367    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.857753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.844643    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:00.858276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.855563    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.858421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.850345    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.865114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.864155    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.868909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.868871    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:00.890663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.884929    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55ft\" (UniqueName: \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:00.890663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.885478    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:00.890663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.887898    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:00.890663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.890215    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttzr\" (UniqueName: \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:00.895273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.894632    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:00.909741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.909706    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:00.911813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.911778    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:00.937399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.936783    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bll8\" (UniqueName: \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:00.937399 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.937163    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:00.945634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.945390    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:00.950182 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.947596    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6cm\" (UniqueName: \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:00.950621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.950552    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:00.957178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.957141    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:00.957346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.957289    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:00.960966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.960931    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5hr\" (UniqueName: \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.961129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.961063    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.961239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.961220    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgcv\" (UniqueName: \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.961468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.961447    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.968953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.968481    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:00.970594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.970565    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:00.997705 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:00.997666    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgcv\" (UniqueName: \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:01.012304 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.010709    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5hr\" (UniqueName: \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:01.064657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.063909    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thc6t\" (UniqueName: \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.064657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.064072    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.080335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.080041    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:01.123066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.122616    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:01.143311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.142691    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:01.166710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.166112    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thc6t\" (UniqueName: \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.166710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.166271    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.167502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.167478    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.172859 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.172809    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:01.177413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.176574    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:01.197889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.193742    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:01.232753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.230958    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:01.269222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.268870    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thc6t\" (UniqueName: \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.322258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.321929    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:01.462329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.462241    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerStarted","Data":"373efceabf15a7df337e7c652151a3696ad47e4d097eeff3d367e38a7a108ad5"}
Oct 30 19:15:01.695041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.695000    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:01.708461 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:01.708384    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a9406_1e09_4f18_8efd_63e6db68afb6.slice/crio-441a349984104b66bf9265e49f8505e04548c8a2363e00f0ac746925621e49c0 WatchSource:0}: Error finding container 441a349984104b66bf9265e49f8505e04548c8a2363e00f0ac746925621e49c0: Status 404 returned error can't find the container with id 441a349984104b66bf9265e49f8505e04548c8a2363e00f0ac746925621e49c0
Oct 30 19:15:01.729162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:01.729125    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:01.745574 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:01.744213    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec7cab6_53dc_406b_9ae6_81386cc82a51.slice/crio-ae1f5238e34204094c84c13cb156ff30723d75273b2b9eaf6fce9a224e522f59 WatchSource:0}: Error finding container ae1f5238e34204094c84c13cb156ff30723d75273b2b9eaf6fce9a224e522f59: Status 404 returned error can't find the container with id ae1f5238e34204094c84c13cb156ff30723d75273b2b9eaf6fce9a224e522f59
Oct 30 19:15:02.033890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.033860    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:02.035833 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:02.035802    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c229b7b_8d99_4441_a5d2_591584b9b780.slice/crio-e3e2c9c22cdd262ed59aade27afb3896131c2121d5add1458981f53def2b1caa WatchSource:0}: Error finding container e3e2c9c22cdd262ed59aade27afb3896131c2121d5add1458981f53def2b1caa: Status 404 returned error can't find the container with id e3e2c9c22cdd262ed59aade27afb3896131c2121d5add1458981f53def2b1caa
Oct 30 19:15:02.043450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.043416    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:02.057712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.057571    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:02.093714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.092813    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:02.134190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.134148    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:02.141988 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:02.141947    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda754bed2_f5ce_4b96_96f2_3a38ba8b264b.slice/crio-10dad1ed149b9e58f012f303204409b886e53df582e895f781d322cb7aa4ef48 WatchSource:0}: Error finding container 10dad1ed149b9e58f012f303204409b886e53df582e895f781d322cb7aa4ef48: Status 404 returned error can't find the container with id 10dad1ed149b9e58f012f303204409b886e53df582e895f781d322cb7aa4ef48
Oct 30 19:15:02.155441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.155363    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:02.157738 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:02.157658    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7baa188_90a2_4a8b_b5fc_2de07b5c7f6d.slice/crio-f8560acf23eaf5724a729012374efdfdf929102a6c35c44d5e8445155327b174 WatchSource:0}: Error finding container f8560acf23eaf5724a729012374efdfdf929102a6c35c44d5e8445155327b174: Status 404 returned error can't find the container with id f8560acf23eaf5724a729012374efdfdf929102a6c35c44d5e8445155327b174
Oct 30 19:15:02.158071 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:15:02.158046    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0107a981_57e3_438d_96ef_bb845a202804.slice/crio-9fbf6c18d92fa55f35fe5f541f1446b0e2e2568450d9fff815df2a7f5bd7c573 WatchSource:0}: Error finding container 9fbf6c18d92fa55f35fe5f541f1446b0e2e2568450d9fff815df2a7f5bd7c573: Status 404 returned error can't find the container with id 9fbf6c18d92fa55f35fe5f541f1446b0e2e2568450d9fff815df2a7f5bd7c573
Oct 30 19:15:02.162712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.162689    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:02.234262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.234220    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:02.263258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.263218    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:02.290673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.290558    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:02.305273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.305216    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:02.321709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.321650    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:02.339978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.339929    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:02.451470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.443849    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:02.549854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.549822    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerStarted","Data":"dc580eb28f90ab3b3543bfeb372f46a0a4b70b6d734ebe203e383c8dca93a061"}
Oct 30 19:15:02.551901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.551870    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerStarted","Data":"1b89d364a6a8072090b128f3f191a7e5da8ab2a2028a42865267255562aa4d7c"}
Oct 30 19:15:02.553878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.553856    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerStarted","Data":"f8560acf23eaf5724a729012374efdfdf929102a6c35c44d5e8445155327b174"}
Oct 30 19:15:02.555540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.555520    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerStarted","Data":"a5543d72016ba503cae71780ce918ad9c9022acf81df09099158ca0fbb1ddb1a"}
Oct 30 19:15:02.557604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.557579    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerStarted","Data":"d9794f2b7d309a346b5ecbef7cb0234202539d1696e1587bc5c6e52ac0cdb984"}
Oct 30 19:15:02.559226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.559198    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerStarted","Data":"ae1f5238e34204094c84c13cb156ff30723d75273b2b9eaf6fce9a224e522f59"}
Oct 30 19:15:02.560920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.560895    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerStarted","Data":"6a3ba8e11a8a8638085f4dad61978fd21842513e4efc965fa720614af02d603f"}
Oct 30 19:15:02.562646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.562623    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerStarted","Data":"e3e2c9c22cdd262ed59aade27afb3896131c2121d5add1458981f53def2b1caa"}
Oct 30 19:15:02.564138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.564113    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerStarted","Data":"ebb6eee15606e7e78124475185e4610e515a4557da57a343d10f48a3a4726b35"}
Oct 30 19:15:02.565844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.565821    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerStarted","Data":"441a349984104b66bf9265e49f8505e04548c8a2363e00f0ac746925621e49c0"}
Oct 30 19:15:02.567645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.567614    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerStarted","Data":"10dad1ed149b9e58f012f303204409b886e53df582e895f781d322cb7aa4ef48"}
Oct 30 19:15:02.569141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.569121    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerStarted","Data":"542cec21abccb1b3b8bff88649ab43744b7f6b39e7dc95066f75c6de4009923e"}
Oct 30 19:15:02.570494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.570471    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerStarted","Data":"f7d3a0b9c491511638746ae6fb370861e5f258888d9752c7737391c20b6f0896"}
Oct 30 19:15:02.573369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.573339    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerStarted","Data":"5749d49720a7e4413b7f2c60360f63cee74b8c834efc7f918f7a5c0a04b162aa"}
Oct 30 19:15:02.574804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.574776    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerStarted","Data":"9fbf6c18d92fa55f35fe5f541f1446b0e2e2568450d9fff815df2a7f5bd7c573"}
Oct 30 19:15:02.576146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.576123    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerStarted","Data":"2651e80a2ab87f457472c4aebd16bb87c336d2db28eebbe9061e2a834a619e24"}
Oct 30 19:15:02.587187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.587097    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" podStartSLOduration=2.587056771 podStartE2EDuration="2.587056771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.572267059 +0000 UTC m=+2522.835732006" watchObservedRunningTime="2026-10-30 19:15:02.587056771 +0000 UTC m=+2522.850521721"
Oct 30 19:15:02.620107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:02.620025    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" podStartSLOduration=2.6200047509999997 podStartE2EDuration="2.620004751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.610210237 +0000 UTC m=+2522.873675197" watchObservedRunningTime="2026-10-30 19:15:02.620004751 +0000 UTC m=+2522.883469704"
Oct 30 19:15:03.604582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.604545    5065 generic.go:356] "Generic (PLEG): container finished" podID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerID="f4ab75a7440f0a8b5119d7571fd13bb05e2b059e5cc57893738573a7214cd058" exitCode=60
Oct 30 19:15:03.605137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.604690    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerDied","Data":"f4ab75a7440f0a8b5119d7571fd13bb05e2b059e5cc57893738573a7214cd058"}
Oct 30 19:15:03.605267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.605246    5065 scope.go:122] "RemoveContainer" containerID="f4ab75a7440f0a8b5119d7571fd13bb05e2b059e5cc57893738573a7214cd058"
Oct 30 19:15:03.611017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.610972    5065 generic.go:356] "Generic (PLEG): container finished" podID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerID="589e08c2a573ccd51cce85fa4427ca81e5444ad3891f6493b22f7fbdb0de9dea" exitCode=60
Oct 30 19:15:03.611252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.611142    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerDied","Data":"589e08c2a573ccd51cce85fa4427ca81e5444ad3891f6493b22f7fbdb0de9dea"}
Oct 30 19:15:03.611563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.611544    5065 scope.go:122] "RemoveContainer" containerID="589e08c2a573ccd51cce85fa4427ca81e5444ad3891f6493b22f7fbdb0de9dea"
Oct 30 19:15:03.613968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.613942    5065 generic.go:356] "Generic (PLEG): container finished" podID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerID="2f93d89297f726b0984ff4150bae26f65b388d5a754c63a2b4edd36f07c39b78" exitCode=60
Oct 30 19:15:03.614153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.614036    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerDied","Data":"2f93d89297f726b0984ff4150bae26f65b388d5a754c63a2b4edd36f07c39b78"}
Oct 30 19:15:03.614893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.614455    5065 scope.go:122] "RemoveContainer" containerID="2f93d89297f726b0984ff4150bae26f65b388d5a754c63a2b4edd36f07c39b78"
Oct 30 19:15:03.617395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.617371    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerStarted","Data":"3419caa0ba16fdcc1eb14c903613043e19a9932d2e0ea4b1699ba29eca373a5b"}
Oct 30 19:15:03.634516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.629559    5065 generic.go:356] "Generic (PLEG): container finished" podID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerID="477ec2b2074c6aa0bfd069f2b88ae65e46ecbdc493a30f3084337e6436a4d84e" exitCode=60
Oct 30 19:15:03.634516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.630191    5065 scope.go:122] "RemoveContainer" containerID="477ec2b2074c6aa0bfd069f2b88ae65e46ecbdc493a30f3084337e6436a4d84e"
Oct 30 19:15:03.634516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.630966    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerDied","Data":"477ec2b2074c6aa0bfd069f2b88ae65e46ecbdc493a30f3084337e6436a4d84e"}
Oct 30 19:15:03.636160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.636118    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerStarted","Data":"00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1"}
Oct 30 19:15:03.639052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.639016    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerStarted","Data":"348d0dda362198b8318e981a5e1a6ea14d42b5d9b9b7c289848876c5184983ae"}
Oct 30 19:15:03.649601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.648617    5065 generic.go:356] "Generic (PLEG): container finished" podID="5021550d-953a-41bb-b1ec-10fec61689f3" containerID="84c92c51ecdc628ca08027333a274e11f0b2d429b3c2c7f0a100a4ce700d0025" exitCode=60
Oct 30 19:15:03.649601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.648710    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerDied","Data":"84c92c51ecdc628ca08027333a274e11f0b2d429b3c2c7f0a100a4ce700d0025"}
Oct 30 19:15:03.649601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.649282    5065 scope.go:122] "RemoveContainer" containerID="84c92c51ecdc628ca08027333a274e11f0b2d429b3c2c7f0a100a4ce700d0025"
Oct 30 19:15:03.653594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.653558    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerStarted","Data":"20dcd38ebf1448f4e0b741242e1f5e6b85a7ba923595450c09a03fd78bcaabd5"}
Oct 30 19:15:03.653741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.653624    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerStarted","Data":"6447c2e03d26c81f7f54e74858d54209fd59d0326f99b71fc298b5334fa1514e"}
Oct 30 19:15:03.656501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.656468    5065 generic.go:356] "Generic (PLEG): container finished" podID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerID="dc580eb28f90ab3b3543bfeb372f46a0a4b70b6d734ebe203e383c8dca93a061" exitCode=60
Oct 30 19:15:03.656685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.656550    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerDied","Data":"dc580eb28f90ab3b3543bfeb372f46a0a4b70b6d734ebe203e383c8dca93a061"}
Oct 30 19:15:03.656987 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.656964    5065 scope.go:122] "RemoveContainer" containerID="dc580eb28f90ab3b3543bfeb372f46a0a4b70b6d734ebe203e383c8dca93a061"
Oct 30 19:15:03.660021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.659963    5065 generic.go:356] "Generic (PLEG): container finished" podID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerID="4b0d24172a512705d8939556c8e0bc497e52cd27593dbeaa20959ee24eb36424" exitCode=60
Oct 30 19:15:03.660452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.660428    5065 scope.go:122] "RemoveContainer" containerID="4b0d24172a512705d8939556c8e0bc497e52cd27593dbeaa20959ee24eb36424"
Oct 30 19:15:03.660559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.660471    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerDied","Data":"4b0d24172a512705d8939556c8e0bc497e52cd27593dbeaa20959ee24eb36424"}
Oct 30 19:15:03.671222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.670818    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerStarted","Data":"906eb047d93d7640b734204ec2b0d0a5a1aa37f30e0c026ef9a34a5a4b235743"}
Oct 30 19:15:03.693349 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.693287    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" podStartSLOduration=3.693271292 podStartE2EDuration="3.693271292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.688849335 +0000 UTC m=+2523.952314302" watchObservedRunningTime="2026-10-30 19:15:03.693271292 +0000 UTC m=+2523.956736239"
Oct 30 19:15:03.791748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.791688    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" podStartSLOduration=3.791654264 podStartE2EDuration="3.791654264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.781057065 +0000 UTC m=+2524.044522013" watchObservedRunningTime="2026-10-30 19:15:03.791654264 +0000 UTC m=+2524.055119213"
Oct 30 19:15:03.812365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.812306    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" podStartSLOduration=3.812284817 podStartE2EDuration="3.812284817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.801285105 +0000 UTC m=+2524.064750070" watchObservedRunningTime="2026-10-30 19:15:03.812284817 +0000 UTC m=+2524.075749766"
Oct 30 19:15:03.826500 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:03.826460    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda754bed2_f5ce_4b96_96f2_3a38ba8b264b.slice/crio-00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:15:03.848882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.848818    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" podStartSLOduration=3.848799275 podStartE2EDuration="3.848799275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.84866295 +0000 UTC m=+2524.112127909" watchObservedRunningTime="2026-10-30 19:15:03.848799275 +0000 UTC m=+2524.112264224"
Oct 30 19:15:03.906766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:03.906624    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" podStartSLOduration=3.906601932 podStartE2EDuration="3.906601932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.903781773 +0000 UTC m=+2524.167246749" watchObservedRunningTime="2026-10-30 19:15:03.906601932 +0000 UTC m=+2524.170066882"
Oct 30 19:15:04.686788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.686747    5065 generic.go:356] "Generic (PLEG): container finished" podID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerID="906eb047d93d7640b734204ec2b0d0a5a1aa37f30e0c026ef9a34a5a4b235743" exitCode=60
Oct 30 19:15:04.687305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.686830    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerDied","Data":"906eb047d93d7640b734204ec2b0d0a5a1aa37f30e0c026ef9a34a5a4b235743"}
Oct 30 19:15:04.687398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.687335    5065 scope.go:122] "RemoveContainer" containerID="906eb047d93d7640b734204ec2b0d0a5a1aa37f30e0c026ef9a34a5a4b235743"
Oct 30 19:15:04.689212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.689186    5065 generic.go:356] "Generic (PLEG): container finished" podID="1be31543-8f9c-47ca-b794-a6000967be6a" containerID="0ac24bc3f876a4be92f248ab0f6372f4e7460277017f80ea372a986970895441" exitCode=60
Oct 30 19:15:04.689335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.689277    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerDied","Data":"0ac24bc3f876a4be92f248ab0f6372f4e7460277017f80ea372a986970895441"}
Oct 30 19:15:04.689686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.689605    5065 scope.go:122] "RemoveContainer" containerID="0ac24bc3f876a4be92f248ab0f6372f4e7460277017f80ea372a986970895441"
Oct 30 19:15:04.692578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.692275    5065 generic.go:356] "Generic (PLEG): container finished" podID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerID="282f5829ecacda24bca9795d1b5ecc54dff25e21f665723997db7d403b5ddec7" exitCode=60
Oct 30 19:15:04.692578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.692440    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerDied","Data":"282f5829ecacda24bca9795d1b5ecc54dff25e21f665723997db7d403b5ddec7"}
Oct 30 19:15:04.692774 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.692728    5065 scope.go:122] "RemoveContainer" containerID="282f5829ecacda24bca9795d1b5ecc54dff25e21f665723997db7d403b5ddec7"
Oct 30 19:15:04.695317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.695297    5065 generic.go:356] "Generic (PLEG): container finished" podID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerID="3419caa0ba16fdcc1eb14c903613043e19a9932d2e0ea4b1699ba29eca373a5b" exitCode=60
Oct 30 19:15:04.695397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.695369    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerDied","Data":"3419caa0ba16fdcc1eb14c903613043e19a9932d2e0ea4b1699ba29eca373a5b"}
Oct 30 19:15:04.695738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.695719    5065 scope.go:122] "RemoveContainer" containerID="3419caa0ba16fdcc1eb14c903613043e19a9932d2e0ea4b1699ba29eca373a5b"
Oct 30 19:15:04.697481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.697456    5065 generic.go:356] "Generic (PLEG): container finished" podID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerID="00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1" exitCode=60
Oct 30 19:15:04.697947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.697931    5065 scope.go:122] "RemoveContainer" containerID="00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1"
Oct 30 19:15:04.698203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.698177    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerDied","Data":"00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1"}
Oct 30 19:15:04.700306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.700277    5065 generic.go:356] "Generic (PLEG): container finished" podID="219ac27f-79b4-4973-a403-4958a49aec9c" containerID="a0bbe5a055168c06411085b9ae6c353bb6329d1da3394b3bb5ff49ea817908cb" exitCode=60
Oct 30 19:15:04.700436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.700364    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerDied","Data":"a0bbe5a055168c06411085b9ae6c353bb6329d1da3394b3bb5ff49ea817908cb"}
Oct 30 19:15:04.700671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.700652    5065 scope.go:122] "RemoveContainer" containerID="a0bbe5a055168c06411085b9ae6c353bb6329d1da3394b3bb5ff49ea817908cb"
Oct 30 19:15:04.702465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.702440    5065 generic.go:356] "Generic (PLEG): container finished" podID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerID="348d0dda362198b8318e981a5e1a6ea14d42b5d9b9b7c289848876c5184983ae" exitCode=60
Oct 30 19:15:04.702585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.702527    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerDied","Data":"348d0dda362198b8318e981a5e1a6ea14d42b5d9b9b7c289848876c5184983ae"}
Oct 30 19:15:04.702817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.702798    5065 scope.go:122] "RemoveContainer" containerID="348d0dda362198b8318e981a5e1a6ea14d42b5d9b9b7c289848876c5184983ae"
Oct 30 19:15:04.704559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.704535    5065 generic.go:356] "Generic (PLEG): container finished" podID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerID="20dcd38ebf1448f4e0b741242e1f5e6b85a7ba923595450c09a03fd78bcaabd5" exitCode=60
Oct 30 19:15:04.704764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.704583    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerDied","Data":"20dcd38ebf1448f4e0b741242e1f5e6b85a7ba923595450c09a03fd78bcaabd5"}
Oct 30 19:15:04.704949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.704927    5065 scope.go:122] "RemoveContainer" containerID="20dcd38ebf1448f4e0b741242e1f5e6b85a7ba923595450c09a03fd78bcaabd5"
Oct 30 19:15:04.708258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.708232    5065 generic.go:356] "Generic (PLEG): container finished" podID="0107a981-57e3-438d-96ef-bb845a202804" containerID="9d6daf8f3aa61c8fce0dac1462bd5f5c62e536dc7332bfb7a3d07d8dd113f115" exitCode=60
Oct 30 19:15:04.708373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.708326    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerDied","Data":"9d6daf8f3aa61c8fce0dac1462bd5f5c62e536dc7332bfb7a3d07d8dd113f115"}
Oct 30 19:15:04.709047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.708616    5065 scope.go:122] "RemoveContainer" containerID="9d6daf8f3aa61c8fce0dac1462bd5f5c62e536dc7332bfb7a3d07d8dd113f115"
Oct 30 19:15:04.711063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.711022    5065 generic.go:356] "Generic (PLEG): container finished" podID="28fad370-0534-487f-aa08-d04982ebc22e" containerID="8cafe3a55aaebb3114f2fe3c289e73e504bd2460365fddd79277b319c4519072" exitCode=60
Oct 30 19:15:04.716161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.711218    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerDied","Data":"8cafe3a55aaebb3114f2fe3c289e73e504bd2460365fddd79277b319c4519072"}
Oct 30 19:15:04.716161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:04.711639    5065 scope.go:122] "RemoveContainer" containerID="8cafe3a55aaebb3114f2fe3c289e73e504bd2460365fddd79277b319c4519072"
Oct 30 19:15:05.040185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.040134    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"]
Oct 30 19:15:05.046028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.045991    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4af195-ccdgf"]
Oct 30 19:15:05.727987 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.727548    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerStarted","Data":"f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"}
Oct 30 19:15:05.730639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.730604    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerStarted","Data":"43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"}
Oct 30 19:15:05.733447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.733405    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerStarted","Data":"8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"}
Oct 30 19:15:05.737435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.737395    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerStarted","Data":"2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"}
Oct 30 19:15:05.740501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.740464    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerStarted","Data":"63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"}
Oct 30 19:15:05.743631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.743594    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerStarted","Data":"5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"}
Oct 30 19:15:05.746699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.746669    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerStarted","Data":"25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"}
Oct 30 19:15:05.751429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.751380    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerStarted","Data":"35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"}
Oct 30 19:15:05.754049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.754013    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerStarted","Data":"5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"}
Oct 30 19:15:05.757700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.757658    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerStarted","Data":"163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"}
Oct 30 19:15:05.761546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.761507    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerStarted","Data":"438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"}
Oct 30 19:15:05.764684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.764648    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerStarted","Data":"f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"}
Oct 30 19:15:05.802550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.802491    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" podStartSLOduration=5.8024738 podStartE2EDuration="5.8024738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.793754198 +0000 UTC m=+2526.057219165" watchObservedRunningTime="2026-10-30 19:15:05.8024738 +0000 UTC m=+2526.065938755"
Oct 30 19:15:05.852868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.852808    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" podStartSLOduration=5.852790391 podStartE2EDuration="5.852790391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.845972327 +0000 UTC m=+2526.109437276" watchObservedRunningTime="2026-10-30 19:15:05.852790391 +0000 UTC m=+2526.116255341"
Oct 30 19:15:05.877185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.877126    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" podStartSLOduration=5.877112121 podStartE2EDuration="5.877112121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.873125843 +0000 UTC m=+2526.136590814" watchObservedRunningTime="2026-10-30 19:15:05.877112121 +0000 UTC m=+2526.140577068"
Oct 30 19:15:05.912660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.912626    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae69d73-d4de-4bf3-b58d-20327c90ce0d" path="/var/lib/kubelet/pods/cae69d73-d4de-4bf3-b58d-20327c90ce0d/volumes"
Oct 30 19:15:05.926207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.926152    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" podStartSLOduration=5.926132723 podStartE2EDuration="5.926132723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.921593923 +0000 UTC m=+2526.185058871" watchObservedRunningTime="2026-10-30 19:15:05.926132723 +0000 UTC m=+2526.189597674"
Oct 30 19:15:05.959434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.959314    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" podStartSLOduration=5.959298432 podStartE2EDuration="5.959298432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.949764525 +0000 UTC m=+2526.213229475" watchObservedRunningTime="2026-10-30 19:15:05.959298432 +0000 UTC m=+2526.222763380"
Oct 30 19:15:05.974300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:05.974248    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" podStartSLOduration=5.974233655 podStartE2EDuration="5.974233655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.973755485 +0000 UTC m=+2526.237220445" watchObservedRunningTime="2026-10-30 19:15:05.974233655 +0000 UTC m=+2526.237698602"
Oct 30 19:15:06.016741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.016687    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" podStartSLOduration=6.016670683 podStartE2EDuration="6.016670683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:06.011545862 +0000 UTC m=+2526.275010830" watchObservedRunningTime="2026-10-30 19:15:06.016670683 +0000 UTC m=+2526.280135630"
Oct 30 19:15:06.054572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.054517    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" podStartSLOduration=6.054499627 podStartE2EDuration="6.054499627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:06.046567486 +0000 UTC m=+2526.310032431" watchObservedRunningTime="2026-10-30 19:15:06.054499627 +0000 UTC m=+2526.317964589"
Oct 30 19:15:06.779395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.779363    5065 generic.go:356] "Generic (PLEG): container finished" podID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerID="f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd" exitCode=60
Oct 30 19:15:06.779918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.779468    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerDied","Data":"f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd"}
Oct 30 19:15:06.779918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.779554    5065 scope.go:122] "RemoveContainer" containerID="348d0dda362198b8318e981a5e1a6ea14d42b5d9b9b7c289848876c5184983ae"
Oct 30 19:15:06.779918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.779863    5065 scope.go:122] "RemoveContainer" containerID="f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd"
Oct 30 19:15:06.780354 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.780323    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4_odh-ai-gateway-infra(951dfe1a-96eb-4dde-b28d-7a951bdb48cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd"
Oct 30 19:15:06.782203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.782164    5065 generic.go:356] "Generic (PLEG): container finished" podID="5021550d-953a-41bb-b1ec-10fec61689f3" containerID="163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8" exitCode=60
Oct 30 19:15:06.782325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.782248    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerDied","Data":"163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"}
Oct 30 19:15:06.782508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.782491    5065 scope.go:122] "RemoveContainer" containerID="163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"
Oct 30 19:15:06.782959 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.782931    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6_odh-ai-gateway-infra(5021550d-953a-41bb-b1ec-10fec61689f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" podUID="5021550d-953a-41bb-b1ec-10fec61689f3"
Oct 30 19:15:06.784717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.784691    5065 generic.go:356] "Generic (PLEG): container finished" podID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerID="438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a" exitCode=60
Oct 30 19:15:06.784908 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.784886    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerDied","Data":"438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"}
Oct 30 19:15:06.785122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.785065    5065 scope.go:122] "RemoveContainer" containerID="438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"
Oct 30 19:15:06.785385 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.785363    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt_odh-ai-gateway-infra(8ba32f64-738c-4165-8b03-0a9fd816d9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea"
Oct 30 19:15:06.787180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.787157    5065 generic.go:356] "Generic (PLEG): container finished" podID="0107a981-57e3-438d-96ef-bb845a202804" containerID="f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3" exitCode=60
Oct 30 19:15:06.787416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.787394    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerDied","Data":"f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"}
Oct 30 19:15:06.787526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.787511    5065 scope.go:122] "RemoveContainer" containerID="f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"
Oct 30 19:15:06.787792 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.787765    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94_odh-ai-gateway-infra(0107a981-57e3-438d-96ef-bb845a202804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" podUID="0107a981-57e3-438d-96ef-bb845a202804"
Oct 30 19:15:06.789594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.789571    5065 generic.go:356] "Generic (PLEG): container finished" podID="28fad370-0534-487f-aa08-d04982ebc22e" containerID="8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50" exitCode=60
Oct 30 19:15:06.789700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.789622    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerDied","Data":"8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50"}
Oct 30 19:15:06.789932 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.789914    5065 scope.go:122] "RemoveContainer" containerID="8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50"
Oct 30 19:15:06.790264 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.790242    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q_odh-ai-gateway-infra(28fad370-0534-487f-aa08-d04982ebc22e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" podUID="28fad370-0534-487f-aa08-d04982ebc22e"
Oct 30 19:15:06.792318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.792299    5065 generic.go:356] "Generic (PLEG): container finished" podID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerID="f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5" exitCode=60
Oct 30 19:15:06.792524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.792408    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerDied","Data":"f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"}
Oct 30 19:15:06.793228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.792736    5065 scope.go:122] "RemoveContainer" containerID="f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"
Oct 30 19:15:06.793228 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.792984    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g_odh-ai-gateway-infra(bc4e2793-5413-40bf-ac7f-6f63c9ce48c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4"
Oct 30 19:15:06.795138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.795073    5065 generic.go:356] "Generic (PLEG): container finished" podID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerID="43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7" exitCode=60
Oct 30 19:15:06.795237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.795193    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerDied","Data":"43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"}
Oct 30 19:15:06.795396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.795378    5065 scope.go:122] "RemoveContainer" containerID="43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"
Oct 30 19:15:06.795672 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.795642    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q_odh-ai-gateway-infra(015c3c4f-3808-4291-b4c3-9b8ab0453512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512"
Oct 30 19:15:06.797271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.797250    5065 generic.go:356] "Generic (PLEG): container finished" podID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerID="8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e" exitCode=60
Oct 30 19:15:06.797376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.797330    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerDied","Data":"8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"}
Oct 30 19:15:06.797541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.797527    5065 scope.go:122] "RemoveContainer" containerID="8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"
Oct 30 19:15:06.797781 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.797761    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp_odh-ai-gateway-infra(f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d"
Oct 30 19:15:06.799494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.799469    5065 generic.go:356] "Generic (PLEG): container finished" podID="1be31543-8f9c-47ca-b794-a6000967be6a" containerID="2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a" exitCode=60
Oct 30 19:15:06.799707 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.799685    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerDied","Data":"2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"}
Oct 30 19:15:06.800266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.799941    5065 scope.go:122] "RemoveContainer" containerID="2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"
Oct 30 19:15:06.800266 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.800254    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5_odh-ai-gateway-infra(1be31543-8f9c-47ca-b794-a6000967be6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" podUID="1be31543-8f9c-47ca-b794-a6000967be6a"
Oct 30 19:15:06.801984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.801963    5065 generic.go:356] "Generic (PLEG): container finished" podID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerID="63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6" exitCode=60
Oct 30 19:15:06.802122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.802026    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerDied","Data":"63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"}
Oct 30 19:15:06.802329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.802313    5065 scope.go:122] "RemoveContainer" containerID="63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"
Oct 30 19:15:06.802593 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.802573    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s_odh-ai-gateway-infra(4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e"
Oct 30 19:15:06.805367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.805346    5065 generic.go:356] "Generic (PLEG): container finished" podID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerID="5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1" exitCode=60
Oct 30 19:15:06.805472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.805428    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerDied","Data":"5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"}
Oct 30 19:15:06.805653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.805637    5065 scope.go:122] "RemoveContainer" containerID="5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"
Oct 30 19:15:06.805917 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.805898    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4_odh-ai-gateway-infra(7ec7cab6-53dc-406b-9ae6-81386cc82a51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51"
Oct 30 19:15:06.807687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.807651    5065 generic.go:356] "Generic (PLEG): container finished" podID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerID="25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d" exitCode=60
Oct 30 19:15:06.807804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.807738    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerDied","Data":"25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"}
Oct 30 19:15:06.807940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.807925    5065 scope.go:122] "RemoveContainer" containerID="25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"
Oct 30 19:15:06.808187 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.808165    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw_odh-ai-gateway-infra(6271310d-be4e-4a1d-a43e-f3b264fc9dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea"
Oct 30 19:15:06.809889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.809869    5065 generic.go:356] "Generic (PLEG): container finished" podID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerID="35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba" exitCode=60
Oct 30 19:15:06.809975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.809945    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerDied","Data":"35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"}
Oct 30 19:15:06.810209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.810192    5065 scope.go:122] "RemoveContainer" containerID="35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"
Oct 30 19:15:06.810447 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.810429    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm_odh-ai-gateway-infra(6c229b7b-8d99-4441-a5d2-591584b9b780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780"
Oct 30 19:15:06.811969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.811950    5065 generic.go:356] "Generic (PLEG): container finished" podID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerID="c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b" exitCode=60
Oct 30 19:15:06.812128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.812023    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerDied","Data":"c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b"}
Oct 30 19:15:06.812242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.812227    5065 scope.go:122] "RemoveContainer" containerID="c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b"
Oct 30 19:15:06.812456 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.812439    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x_odh-ai-gateway-infra(104c4ab4-75c5-413e-b338-850ec7f0b2d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4"
Oct 30 19:15:06.814010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.813989    5065 generic.go:356] "Generic (PLEG): container finished" podID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerID="5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184" exitCode=60
Oct 30 19:15:06.814134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.814058    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerDied","Data":"5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"}
Oct 30 19:15:06.814283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.814268    5065 scope.go:122] "RemoveContainer" containerID="5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"
Oct 30 19:15:06.814527 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.814506    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs_odh-ai-gateway-infra(486a9406-1e09-4f18-8efd-63e6db68afb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6"
Oct 30 19:15:06.815987 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.815965    5065 generic.go:356] "Generic (PLEG): container finished" podID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerID="18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e" exitCode=60
Oct 30 19:15:06.816115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.816049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerDied","Data":"18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e"}
Oct 30 19:15:06.816310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.816290    5065 scope.go:122] "RemoveContainer" containerID="18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e"
Oct 30 19:15:06.816609 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.816560    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl_odh-ai-gateway-infra(a754bed2-f5ce-4b96-96f2-3a38ba8b264b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b"
Oct 30 19:15:06.818126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.818104    5065 generic.go:356] "Generic (PLEG): container finished" podID="219ac27f-79b4-4973-a403-4958a49aec9c" containerID="25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008" exitCode=60
Oct 30 19:15:06.818203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.818139    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerDied","Data":"25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008"}
Oct 30 19:15:06.818407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.818391    5065 scope.go:122] "RemoveContainer" containerID="25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008"
Oct 30 19:15:06.818685 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:06.818663    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq_odh-ai-gateway-infra(219ac27f-79b4-4973-a403-4958a49aec9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" podUID="219ac27f-79b4-4973-a403-4958a49aec9c"
Oct 30 19:15:06.902381 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.902360    5065 scope.go:122] "RemoveContainer" containerID="84c92c51ecdc628ca08027333a274e11f0b2d429b3c2c7f0a100a4ce700d0025"
Oct 30 19:15:06.929843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.929803    5065 scope.go:122] "RemoveContainer" containerID="20dcd38ebf1448f4e0b741242e1f5e6b85a7ba923595450c09a03fd78bcaabd5"
Oct 30 19:15:06.947326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.947307    5065 scope.go:122] "RemoveContainer" containerID="9d6daf8f3aa61c8fce0dac1462bd5f5c62e536dc7332bfb7a3d07d8dd113f115"
Oct 30 19:15:06.967611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.967590    5065 scope.go:122] "RemoveContainer" containerID="8cafe3a55aaebb3114f2fe3c289e73e504bd2460365fddd79277b319c4519072"
Oct 30 19:15:06.985584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:06.985562    5065 scope.go:122] "RemoveContainer" containerID="dc580eb28f90ab3b3543bfeb372f46a0a4b70b6d734ebe203e383c8dca93a061"
Oct 30 19:15:07.006825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.005909    5065 scope.go:122] "RemoveContainer" containerID="4b0d24172a512705d8939556c8e0bc497e52cd27593dbeaa20959ee24eb36424"
Oct 30 19:15:07.026596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.026571    5065 scope.go:122] "RemoveContainer" containerID="906eb047d93d7640b734204ec2b0d0a5a1aa37f30e0c026ef9a34a5a4b235743"
Oct 30 19:15:07.045675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.045655    5065 scope.go:122] "RemoveContainer" containerID="0ac24bc3f876a4be92f248ab0f6372f4e7460277017f80ea372a986970895441"
Oct 30 19:15:07.061979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.061957    5065 scope.go:122] "RemoveContainer" containerID="282f5829ecacda24bca9795d1b5ecc54dff25e21f665723997db7d403b5ddec7"
Oct 30 19:15:07.078429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.078406    5065 scope.go:122] "RemoveContainer" containerID="f4ab75a7440f0a8b5119d7571fd13bb05e2b059e5cc57893738573a7214cd058"
Oct 30 19:15:07.110793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.110768    5065 scope.go:122] "RemoveContainer" containerID="589e08c2a573ccd51cce85fa4427ca81e5444ad3891f6493b22f7fbdb0de9dea"
Oct 30 19:15:07.126725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.126705    5065 scope.go:122] "RemoveContainer" containerID="2f93d89297f726b0984ff4150bae26f65b388d5a754c63a2b4edd36f07c39b78"
Oct 30 19:15:07.234501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.229272    5065 scope.go:122] "RemoveContainer" containerID="3419caa0ba16fdcc1eb14c903613043e19a9932d2e0ea4b1699ba29eca373a5b"
Oct 30 19:15:07.254302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.254282    5065 scope.go:122] "RemoveContainer" containerID="477ec2b2074c6aa0bfd069f2b88ae65e46ecbdc493a30f3084337e6436a4d84e"
Oct 30 19:15:07.270657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.270636    5065 scope.go:122] "RemoveContainer" containerID="00cdaca54b614bc0b45287ab1afc707ddba4e1c406b52202cf3dcaf1478e8da1"
Oct 30 19:15:07.292298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.292274    5065 scope.go:122] "RemoveContainer" containerID="a0bbe5a055168c06411085b9ae6c353bb6329d1da3394b3bb5ff49ea817908cb"
Oct 30 19:15:07.832585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.832555    5065 scope.go:122] "RemoveContainer" containerID="43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"
Oct 30 19:15:07.833033 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.832962    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q_odh-ai-gateway-infra(015c3c4f-3808-4291-b4c3-9b8ab0453512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512"
Oct 30 19:15:07.834365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.834344    5065 scope.go:122] "RemoveContainer" containerID="8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"
Oct 30 19:15:07.834651 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.834630    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp_odh-ai-gateway-infra(f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d"
Oct 30 19:15:07.836477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.836450    5065 scope.go:122] "RemoveContainer" containerID="2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"
Oct 30 19:15:07.836746 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.836716    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5_odh-ai-gateway-infra(1be31543-8f9c-47ca-b794-a6000967be6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" podUID="1be31543-8f9c-47ca-b794-a6000967be6a"
Oct 30 19:15:07.838569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.838548    5065 scope.go:122] "RemoveContainer" containerID="63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"
Oct 30 19:15:07.838824 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.838802    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s_odh-ai-gateway-infra(4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e"
Oct 30 19:15:07.840848 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.840830    5065 scope.go:122] "RemoveContainer" containerID="5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"
Oct 30 19:15:07.841188 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.841166    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4_odh-ai-gateway-infra(7ec7cab6-53dc-406b-9ae6-81386cc82a51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51"
Oct 30 19:15:07.842567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.842547    5065 scope.go:122] "RemoveContainer" containerID="25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"
Oct 30 19:15:07.842821 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.842799    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw_odh-ai-gateway-infra(6271310d-be4e-4a1d-a43e-f3b264fc9dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea"
Oct 30 19:15:07.844477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.844459    5065 scope.go:122] "RemoveContainer" containerID="35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"
Oct 30 19:15:07.844672 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.844652    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm_odh-ai-gateway-infra(6c229b7b-8d99-4441-a5d2-591584b9b780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780"
Oct 30 19:15:07.846418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.846395    5065 scope.go:122] "RemoveContainer" containerID="c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b"
Oct 30 19:15:07.846695 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.846666    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x_odh-ai-gateway-infra(104c4ab4-75c5-413e-b338-850ec7f0b2d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4"
Oct 30 19:15:07.848284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.848267    5065 scope.go:122] "RemoveContainer" containerID="5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"
Oct 30 19:15:07.848504 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.848485    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs_odh-ai-gateway-infra(486a9406-1e09-4f18-8efd-63e6db68afb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6"
Oct 30 19:15:07.849940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.849922    5065 scope.go:122] "RemoveContainer" containerID="18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e"
Oct 30 19:15:07.850193 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.850166    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl_odh-ai-gateway-infra(a754bed2-f5ce-4b96-96f2-3a38ba8b264b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b"
Oct 30 19:15:07.868047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.868020    5065 scope.go:122] "RemoveContainer" containerID="25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008"
Oct 30 19:15:07.868442 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.868393    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq_odh-ai-gateway-infra(219ac27f-79b4-4973-a403-4958a49aec9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" podUID="219ac27f-79b4-4973-a403-4958a49aec9c"
Oct 30 19:15:07.870568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.870545    5065 scope.go:122] "RemoveContainer" containerID="f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd"
Oct 30 19:15:07.870856 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.870832    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4_odh-ai-gateway-infra(951dfe1a-96eb-4dde-b28d-7a951bdb48cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd"
Oct 30 19:15:07.872745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.872723    5065 scope.go:122] "RemoveContainer" containerID="163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"
Oct 30 19:15:07.872978 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.872950    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6_odh-ai-gateway-infra(5021550d-953a-41bb-b1ec-10fec61689f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" podUID="5021550d-953a-41bb-b1ec-10fec61689f3"
Oct 30 19:15:07.874680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.874662    5065 scope.go:122] "RemoveContainer" containerID="438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"
Oct 30 19:15:07.874938 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.874909    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt_odh-ai-gateway-infra(8ba32f64-738c-4165-8b03-0a9fd816d9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea"
Oct 30 19:15:07.876641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.876618    5065 scope.go:122] "RemoveContainer" containerID="f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"
Oct 30 19:15:07.876872 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.876853    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94_odh-ai-gateway-infra(0107a981-57e3-438d-96ef-bb845a202804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" podUID="0107a981-57e3-438d-96ef-bb845a202804"
Oct 30 19:15:07.878972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.878952    5065 scope.go:122] "RemoveContainer" containerID="8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50"
Oct 30 19:15:07.879206 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.879187    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q_odh-ai-gateway-infra(28fad370-0534-487f-aa08-d04982ebc22e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" podUID="28fad370-0534-487f-aa08-d04982ebc22e"
Oct 30 19:15:07.881256 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:07.881234    5065 scope.go:122] "RemoveContainer" containerID="f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"
Oct 30 19:15:07.881548 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:07.881527    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g_odh-ai-gateway-infra(bc4e2793-5413-40bf-ac7f-6f63c9ce48c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4"
Oct 30 19:15:14.575404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:14.575373    5065 scope.go:122] "RemoveContainer" containerID="7a12c472e1ec696963ee44cb73ff35c831179fd5b6366a10811a4f2d02499110"
Oct 30 19:15:16.109242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:16.098160    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"]
Oct 30 19:15:16.109798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:16.109780    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e25053-pf8x4"]
Oct 30 19:15:16.882813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:16.882770    5065 scope.go:122] "RemoveContainer" containerID="5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"
Oct 30 19:15:16.882976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:16.882909    5065 scope.go:122] "RemoveContainer" containerID="2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"
Oct 30 19:15:16.882976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:16.882968    5065 scope.go:122] "RemoveContainer" containerID="163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"
Oct 30 19:15:17.886078 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:17.886046    5065 scope.go:122] "RemoveContainer" containerID="f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd"
Oct 30 19:15:17.887863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:17.887829    5065 scope.go:122] "RemoveContainer" containerID="25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008"
Oct 30 19:15:17.931536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:17.926907    5065 scope.go:122] "RemoveContainer" containerID="25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"
Oct 30 19:15:17.975153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:17.975123    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a4bb9f-94f8-4052-8bc8-4d5298fe97e9" path="/var/lib/kubelet/pods/57a4bb9f-94f8-4052-8bc8-4d5298fe97e9/volumes"
Oct 30 19:15:18.086313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.084443    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerStarted","Data":"56c472acc465cbbaff2ecbd0c4e08dde224fa3cd099166de600675fe881f4dcd"}
Oct 30 19:15:18.090826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.088059    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerStarted","Data":"d16f327699467c5d1044b61444eeb8715aecd5bb349bd5542df1d7ee2eae646a"}
Oct 30 19:15:18.090938 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.090846    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerStarted","Data":"1fd3981c51f3ad4e6ab7642ee1758b6bb1f28b8112d3aa01a11870198abf7593"}
Oct 30 19:15:18.904833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.901749    5065 scope.go:122] "RemoveContainer" containerID="5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"
Oct 30 19:15:18.932239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.924830    5065 scope.go:122] "RemoveContainer" containerID="18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e"
Oct 30 19:15:18.932239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.926028    5065 scope.go:122] "RemoveContainer" containerID="c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b"
Oct 30 19:15:18.932239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:18.926907    5065 scope.go:122] "RemoveContainer" containerID="43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"
Oct 30 19:15:19.107961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.107927    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerStarted","Data":"d2fd411f3d0ef78c8688f98cb2ae56acfadebb17689b35dca28198ed1531e4b7"}
Oct 30 19:15:19.110477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.110447    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerStarted","Data":"6107456683f7f20138723b32031f24479857f82b81327b06e10893a99ef52d26"}
Oct 30 19:15:19.112462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.112438    5065 generic.go:356] "Generic (PLEG): container finished" podID="5021550d-953a-41bb-b1ec-10fec61689f3" containerID="1fd3981c51f3ad4e6ab7642ee1758b6bb1f28b8112d3aa01a11870198abf7593" exitCode=60
Oct 30 19:15:19.112570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.112506    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerDied","Data":"1fd3981c51f3ad4e6ab7642ee1758b6bb1f28b8112d3aa01a11870198abf7593"}
Oct 30 19:15:19.112570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.112564    5065 scope.go:122] "RemoveContainer" containerID="163cb1f3a9fdb70fc620976ead628260230cac3461d546571de697658b7513e8"
Oct 30 19:15:19.112984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.112963    5065 scope.go:122] "RemoveContainer" containerID="1fd3981c51f3ad4e6ab7642ee1758b6bb1f28b8112d3aa01a11870198abf7593"
Oct 30 19:15:19.113321 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:19.113298    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6_odh-ai-gateway-infra(5021550d-953a-41bb-b1ec-10fec61689f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" podUID="5021550d-953a-41bb-b1ec-10fec61689f3"
Oct 30 19:15:19.116526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.116502    5065 generic.go:356] "Generic (PLEG): container finished" podID="1be31543-8f9c-47ca-b794-a6000967be6a" containerID="56c472acc465cbbaff2ecbd0c4e08dde224fa3cd099166de600675fe881f4dcd" exitCode=60
Oct 30 19:15:19.116628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.116571    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerDied","Data":"56c472acc465cbbaff2ecbd0c4e08dde224fa3cd099166de600675fe881f4dcd"}
Oct 30 19:15:19.116922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.116905    5065 scope.go:122] "RemoveContainer" containerID="56c472acc465cbbaff2ecbd0c4e08dde224fa3cd099166de600675fe881f4dcd"
Oct 30 19:15:19.117264 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:19.117236    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5_odh-ai-gateway-infra(1be31543-8f9c-47ca-b794-a6000967be6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" podUID="1be31543-8f9c-47ca-b794-a6000967be6a"
Oct 30 19:15:19.119272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.119250    5065 generic.go:356] "Generic (PLEG): container finished" podID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerID="d16f327699467c5d1044b61444eeb8715aecd5bb349bd5542df1d7ee2eae646a" exitCode=60
Oct 30 19:15:19.119377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.119327    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerDied","Data":"d16f327699467c5d1044b61444eeb8715aecd5bb349bd5542df1d7ee2eae646a"}
Oct 30 19:15:19.119692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.119664    5065 scope.go:122] "RemoveContainer" containerID="d16f327699467c5d1044b61444eeb8715aecd5bb349bd5542df1d7ee2eae646a"
Oct 30 19:15:19.120135 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:19.119993    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4_odh-ai-gateway-infra(7ec7cab6-53dc-406b-9ae6-81386cc82a51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51"
Oct 30 19:15:19.121955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.121929    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerStarted","Data":"12df46ce8f20fdea406c9bb24d2c9c682e4506f6c45612b955e813cc41e79d1f"}
Oct 30 19:15:19.128278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.128257    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:19.131027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.131007    5065 scope.go:122] "RemoveContainer" containerID="2dddeccb8fbcd33f9ec8a9c94153fdacee37f112f2aa015d47fd6f14d34cf42a"
Oct 30 19:15:19.140072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.140022    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" podStartSLOduration=19.140011188 podStartE2EDuration="19.140011188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.133284916 +0000 UTC m=+2539.396749880" watchObservedRunningTime="2026-10-30 19:15:19.140011188 +0000 UTC m=+2539.403476136"
Oct 30 19:15:19.147268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.147250    5065 scope.go:122] "RemoveContainer" containerID="5745e112779eb09293d4b17753c61eb3620702bda864159dc049650bf8a933d1"
Oct 30 19:15:19.155326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.155281    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:19.172689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.172667    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:19.971738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.971650    5065 scope.go:122] "RemoveContainer" containerID="63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"
Oct 30 19:15:19.972242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.972222    5065 scope.go:122] "RemoveContainer" containerID="8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"
Oct 30 19:15:19.972349 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.972334    5065 scope.go:122] "RemoveContainer" containerID="35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"
Oct 30 19:15:19.972977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.972957    5065 scope.go:122] "RemoveContainer" containerID="f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"
Oct 30 19:15:19.973058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:19.973044    5065 scope.go:122] "RemoveContainer" containerID="438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"
Oct 30 19:15:20.141371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.141337    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerStarted","Data":"633279bc624cfe852bfd683a6c737d5e37c951c4c02d5b4cd21c87b0a8ec8880"}
Oct 30 19:15:20.144803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.144776    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerStarted","Data":"907b859958bc484ffd1595ae192db0e22b19ab64e3c12b1e19ceced79191900a"}
Oct 30 19:15:20.147541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.147502    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerStarted","Data":"8abe30072bc560f2ba55efd84825b8a9e7500266b8f5d26e9eb5186f28801be6"}
Oct 30 19:15:20.150420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.150397    5065 generic.go:356] "Generic (PLEG): container finished" podID="219ac27f-79b4-4973-a403-4958a49aec9c" containerID="d2fd411f3d0ef78c8688f98cb2ae56acfadebb17689b35dca28198ed1531e4b7" exitCode=60
Oct 30 19:15:20.150501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.150469    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerDied","Data":"d2fd411f3d0ef78c8688f98cb2ae56acfadebb17689b35dca28198ed1531e4b7"}
Oct 30 19:15:20.150562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.150525    5065 scope.go:122] "RemoveContainer" containerID="25ea64850538b163dfd2d54394423d6731853d700a048ca8970bf54e18a8d008"
Oct 30 19:15:20.150921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.150899    5065 scope.go:122] "RemoveContainer" containerID="d2fd411f3d0ef78c8688f98cb2ae56acfadebb17689b35dca28198ed1531e4b7"
Oct 30 19:15:20.151327 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:20.151298    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq_odh-ai-gateway-infra(219ac27f-79b4-4973-a403-4958a49aec9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" podUID="219ac27f-79b4-4973-a403-4958a49aec9c"
Oct 30 19:15:20.153173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.153150    5065 generic.go:356] "Generic (PLEG): container finished" podID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerID="6107456683f7f20138723b32031f24479857f82b81327b06e10893a99ef52d26" exitCode=60
Oct 30 19:15:20.153293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.153250    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerDied","Data":"6107456683f7f20138723b32031f24479857f82b81327b06e10893a99ef52d26"}
Oct 30 19:15:20.153594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.153574    5065 scope.go:122] "RemoveContainer" containerID="6107456683f7f20138723b32031f24479857f82b81327b06e10893a99ef52d26"
Oct 30 19:15:20.153826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.153806    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:20.153902 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:20.153869    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4_odh-ai-gateway-infra(951dfe1a-96eb-4dde-b28d-7a951bdb48cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd"
Oct 30 19:15:20.158053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.158029    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerStarted","Data":"580cf37cd99ee8c4000018ebf783cb81db677df1aabf44baa07664df35c14da2"}
Oct 30 19:15:20.168251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.168226    5065 generic.go:356] "Generic (PLEG): container finished" podID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerID="12df46ce8f20fdea406c9bb24d2c9c682e4506f6c45612b955e813cc41e79d1f" exitCode=60
Oct 30 19:15:20.168356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.168302    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerDied","Data":"12df46ce8f20fdea406c9bb24d2c9c682e4506f6c45612b955e813cc41e79d1f"}
Oct 30 19:15:20.168680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.168660    5065 scope.go:122] "RemoveContainer" containerID="12df46ce8f20fdea406c9bb24d2c9c682e4506f6c45612b955e813cc41e79d1f"
Oct 30 19:15:20.169189 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:20.169164    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw_odh-ai-gateway-infra(6271310d-be4e-4a1d-a43e-f3b264fc9dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea"
Oct 30 19:15:20.175431 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.175328    5065 scope.go:122] "RemoveContainer" containerID="f6671769df99abbfe1f208c1c19d98e0941fcef380f4ae5fa9e501bd580a73bd"
Oct 30 19:15:20.212077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.209017    5065 scope.go:122] "RemoveContainer" containerID="25c8e58a17629bd84e8e30a154aba1fb2562d3819ab0e1a069264a8207d8e12d"
Oct 30 19:15:20.245410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.245380    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:20.261258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.261230    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:20.695008 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.694974    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:20.808289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.808262    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:20.815827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.815800    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:20.882749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.875010    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\" (UniqueName: \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\") pod \"5021550d-953a-41bb-b1ec-10fec61689f3\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") "
Oct 30 19:15:20.882749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.875188    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\") pod \"5021550d-953a-41bb-b1ec-10fec61689f3\" (UID: \"5021550d-953a-41bb-b1ec-10fec61689f3\") "
Oct 30 19:15:20.908974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.887526    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca" pod "5021550d-953a-41bb-b1ec-10fec61689f3" (UID: "5021550d-953a-41bb-b1ec-10fec61689f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.908974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.889238    5065 scope.go:122] "RemoveContainer" containerID="8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50"
Oct 30 19:15:20.945359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.913268    5065 scope.go:122] "RemoveContainer" containerID="f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"
Oct 30 19:15:20.989744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.989714    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\" (UniqueName: \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\") pod \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") "
Oct 30 19:15:20.990175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.989965    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\") pod \"1be31543-8f9c-47ca-b794-a6000967be6a\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") "
Oct 30 19:15:20.990175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.990154    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\") pod \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\" (UID: \"7ec7cab6-53dc-406b-9ae6-81386cc82a51\") "
Oct 30 19:15:20.990268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.990258    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\" (UniqueName: \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\") pod \"1be31543-8f9c-47ca-b794-a6000967be6a\" (UID: \"1be31543-8f9c-47ca-b794-a6000967be6a\") "
Oct 30 19:15:20.990950 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.990931    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021550d-953a-41bb-b1ec-10fec61689f3-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.992673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.992648    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca" pod "1be31543-8f9c-47ca-b794-a6000967be6a" (UID: "1be31543-8f9c-47ca-b794-a6000967be6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.993468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.993449    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca" pod "7ec7cab6-53dc-406b-9ae6-81386cc82a51" (UID: "7ec7cab6-53dc-406b-9ae6-81386cc82a51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.994257 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.994160    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx" pod "7ec7cab6-53dc-406b-9ae6-81386cc82a51" (UID: "7ec7cab6-53dc-406b-9ae6-81386cc82a51"). InnerVolumeSpecName "kube-api-access-sf2rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.994952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.994930    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm" pod "1be31543-8f9c-47ca-b794-a6000967be6a" (UID: "1be31543-8f9c-47ca-b794-a6000967be6a"). InnerVolumeSpecName "kube-api-access-kz6cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.998023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:20.997999    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z" pod "5021550d-953a-41bb-b1ec-10fec61689f3" (UID: "5021550d-953a-41bb-b1ec-10fec61689f3"). InnerVolumeSpecName "kube-api-access-v2d5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.092393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.092359    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be31543-8f9c-47ca-b794-a6000967be6a-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.092393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.092387    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2d5z\" (UniqueName: \"kubernetes.io/projected/5021550d-953a-41bb-b1ec-10fec61689f3-kube-api-access-v2d5z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.092565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.092400    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec7cab6-53dc-406b-9ae6-81386cc82a51-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.092565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.092409    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6cm\" (UniqueName: \"kubernetes.io/projected/1be31543-8f9c-47ca-b794-a6000967be6a-kube-api-access-kz6cm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.092565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.092419    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf2rx\" (UniqueName: \"kubernetes.io/projected/7ec7cab6-53dc-406b-9ae6-81386cc82a51-kube-api-access-sf2rx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.198384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.198348    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:21.201982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.201956    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerStarted","Data":"817fc912be731088b9ee773e7341e2c49c1fcaaba5d8c649b507a4a2a8d83584"}
Oct 30 19:15:21.205423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.205396    5065 generic.go:356] "Generic (PLEG): container finished" podID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerID="580cf37cd99ee8c4000018ebf783cb81db677df1aabf44baa07664df35c14da2" exitCode=60
Oct 30 19:15:21.205600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.205474    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerDied","Data":"580cf37cd99ee8c4000018ebf783cb81db677df1aabf44baa07664df35c14da2"}
Oct 30 19:15:21.205600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.205522    5065 scope.go:122] "RemoveContainer" containerID="43949ecd5b4285f2d6b9833869a5a7c84641a953a2b14a276ced854a736d9ee7"
Oct 30 19:15:21.205966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.205948    5065 scope.go:122] "RemoveContainer" containerID="580cf37cd99ee8c4000018ebf783cb81db677df1aabf44baa07664df35c14da2"
Oct 30 19:15:21.206404 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:21.206372    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q_odh-ai-gateway-infra(015c3c4f-3808-4291-b4c3-9b8ab0453512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512"
Oct 30 19:15:21.210165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.210144    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerStarted","Data":"768075306cdf97eb67cb0dd383595f7101f0f525206464c8eceec1a14fe11e1f"}
Oct 30 19:15:21.213848 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.213828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5" event={"ID":"1be31543-8f9c-47ca-b794-a6000967be6a","Type":"ContainerDied","Data":"a5543d72016ba503cae71780ce918ad9c9022acf81df09099158ca0fbb1ddb1a"}
Oct 30 19:15:21.213944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.213854    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"
Oct 30 19:15:21.215896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.215873    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerStarted","Data":"f3fa8dcc010331287d8d181cb148147f6db4e6fc56b831397cadeed6cec6b4df"}
Oct 30 19:15:21.223731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.223703    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4" event={"ID":"7ec7cab6-53dc-406b-9ae6-81386cc82a51","Type":"ContainerDied","Data":"ae1f5238e34204094c84c13cb156ff30723d75273b2b9eaf6fce9a224e522f59"}
Oct 30 19:15:21.223833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.223784    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"
Oct 30 19:15:21.229956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.229896    5065 scope.go:122] "RemoveContainer" containerID="56c472acc465cbbaff2ecbd0c4e08dde224fa3cd099166de600675fe881f4dcd"
Oct 30 19:15:21.235477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.235393    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:21.236219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.235906    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerStarted","Data":"0da54a0dd86c16aaac32147afdee8c94e70831d78ee68e4745a39069ddd7722d"}
Oct 30 19:15:21.238584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.238564    5065 generic.go:356] "Generic (PLEG): container finished" podID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerID="633279bc624cfe852bfd683a6c737d5e37c951c4c02d5b4cd21c87b0a8ec8880" exitCode=60
Oct 30 19:15:21.238661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.238616    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerDied","Data":"633279bc624cfe852bfd683a6c737d5e37c951c4c02d5b4cd21c87b0a8ec8880"}
Oct 30 19:15:21.240803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.240782    5065 generic.go:356] "Generic (PLEG): container finished" podID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerID="907b859958bc484ffd1595ae192db0e22b19ab64e3c12b1e19ceced79191900a" exitCode=60
Oct 30 19:15:21.241003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.240983    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerDied","Data":"907b859958bc484ffd1595ae192db0e22b19ab64e3c12b1e19ceced79191900a"}
Oct 30 19:15:21.241562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.241541    5065 scope.go:122] "RemoveContainer" containerID="907b859958bc484ffd1595ae192db0e22b19ab64e3c12b1e19ceced79191900a"
Oct 30 19:15:21.242010 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:21.241985    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs_odh-ai-gateway-infra(486a9406-1e09-4f18-8efd-63e6db68afb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6"
Oct 30 19:15:21.243391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.243190    5065 generic.go:356] "Generic (PLEG): container finished" podID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerID="8abe30072bc560f2ba55efd84825b8a9e7500266b8f5d26e9eb5186f28801be6" exitCode=60
Oct 30 19:15:21.243391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.243269    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerDied","Data":"8abe30072bc560f2ba55efd84825b8a9e7500266b8f5d26e9eb5186f28801be6"}
Oct 30 19:15:21.246484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.246467    5065 scope.go:122] "RemoveContainer" containerID="d16f327699467c5d1044b61444eeb8715aecd5bb349bd5542df1d7ee2eae646a"
Oct 30 19:15:21.253611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.253593    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"
Oct 30 19:15:21.253864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.253609    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6" event={"ID":"5021550d-953a-41bb-b1ec-10fec61689f3","Type":"ContainerDied","Data":"5749d49720a7e4413b7f2c60360f63cee74b8c834efc7f918f7a5c0a04b162aa"}
Oct 30 19:15:21.261314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.261291    5065 scope.go:122] "RemoveContainer" containerID="c02f4cb87628413227fec6b5e5373cb18dfdc7703dca7057eab75cb686be173b"
Oct 30 19:15:21.261946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.261917    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerStarted","Data":"c2f26694b7715f06dae68cb44a60de97239d60cfcf19c3a5a1f8a0ff3e94f7b1"}
Oct 30 19:15:21.283236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.283218    5065 scope.go:122] "RemoveContainer" containerID="5f10f4ae8469e75bcc830a555aa38e4b2974f16b7bc21111258c03888147f184"
Oct 30 19:15:21.305352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.305295    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:21.329000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.326808    5065 scope.go:122] "RemoveContainer" containerID="18263d95b1ceb1bb08f9fa865d570208547ddbb48413d245962e7521f26aea0e"
Oct 30 19:15:21.331570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.331475    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:21.344822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.344710    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-86c150-29889795-z76w5"]
Oct 30 19:15:21.352926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.352833    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:21.359584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.359178    5065 scope.go:122] "RemoveContainer" containerID="1fd3981c51f3ad4e6ab7642ee1758b6bb1f28b8112d3aa01a11870198abf7593"
Oct 30 19:15:21.543887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.543850    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:21.554811 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.554791    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e8c554-29889795-vgxb4"]
Oct 30 19:15:21.567040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.565158    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:21.577616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.575961    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689c23-29889795-4lgq6"]
Oct 30 19:15:21.763921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.763898    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:21.767687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.767620    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:21.778684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.778662    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:21.808763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:21.808690    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:22.025308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025276    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\" (UniqueName: \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\") pod \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025342    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\") pod \"219ac27f-79b4-4973-a403-4958a49aec9c\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025499    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\" (UniqueName: \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\") pod \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025553    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\" (UniqueName: \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\") pod \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025674    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\") pod \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\" (UID: \"104c4ab4-75c5-413e-b338-850ec7f0b2d4\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025736    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\") pod \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\" (UID: \"6271310d-be4e-4a1d-a43e-f3b264fc9dea\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025757    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\" (UniqueName: \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\") pod \"219ac27f-79b4-4973-a403-4958a49aec9c\" (UID: \"219ac27f-79b4-4973-a403-4958a49aec9c\") "
Oct 30 19:15:22.025804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.025788    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\") pod \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\" (UID: \"951dfe1a-96eb-4dde-b28d-7a951bdb48cd\") "
Oct 30 19:15:22.031200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.031168    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595" pod "6271310d-be4e-4a1d-a43e-f3b264fc9dea" (UID: "6271310d-be4e-4a1d-a43e-f3b264fc9dea"). InnerVolumeSpecName "kube-api-access-zp595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.031532 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.031509    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca" pod "219ac27f-79b4-4973-a403-4958a49aec9c" (UID: "219ac27f-79b4-4973-a403-4958a49aec9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.033385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.033357    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" path="/var/lib/kubelet/pods/1be31543-8f9c-47ca-b794-a6000967be6a/volumes"
Oct 30 19:15:22.034247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.034223    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" path="/var/lib/kubelet/pods/5021550d-953a-41bb-b1ec-10fec61689f3/volumes"
Oct 30 19:15:22.034844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.034826    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" path="/var/lib/kubelet/pods/7ec7cab6-53dc-406b-9ae6-81386cc82a51/volumes"
Oct 30 19:15:22.035834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.035804    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t" pod "104c4ab4-75c5-413e-b338-850ec7f0b2d4" (UID: "104c4ab4-75c5-413e-b338-850ec7f0b2d4"). InnerVolumeSpecName "kube-api-access-thc6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.037204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.037180    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca" pod "104c4ab4-75c5-413e-b338-850ec7f0b2d4" (UID: "104c4ab4-75c5-413e-b338-850ec7f0b2d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.037674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.037653    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca" pod "6271310d-be4e-4a1d-a43e-f3b264fc9dea" (UID: "6271310d-be4e-4a1d-a43e-f3b264fc9dea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.038343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.038318    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv" pod "951dfe1a-96eb-4dde-b28d-7a951bdb48cd" (UID: "951dfe1a-96eb-4dde-b28d-7a951bdb48cd"). InnerVolumeSpecName "kube-api-access-9xgcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.038761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.038739    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca" pod "951dfe1a-96eb-4dde-b28d-7a951bdb48cd" (UID: "951dfe1a-96eb-4dde-b28d-7a951bdb48cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.041336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.041315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8" pod "219ac27f-79b4-4973-a403-4958a49aec9c" (UID: "219ac27f-79b4-4973-a403-4958a49aec9c"). InnerVolumeSpecName "kube-api-access-8bll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129240    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp595\" (UniqueName: \"kubernetes.io/projected/6271310d-be4e-4a1d-a43e-f3b264fc9dea-kube-api-access-zp595\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129271    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219ac27f-79b4-4973-a403-4958a49aec9c-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129288    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thc6t\" (UniqueName: \"kubernetes.io/projected/104c4ab4-75c5-413e-b338-850ec7f0b2d4-kube-api-access-thc6t\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129304    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgcv\" (UniqueName: \"kubernetes.io/projected/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-kube-api-access-9xgcv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129317    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104c4ab4-75c5-413e-b338-850ec7f0b2d4-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129330    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6271310d-be4e-4a1d-a43e-f3b264fc9dea-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129345    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bll8\" (UniqueName: \"kubernetes.io/projected/219ac27f-79b4-4973-a403-4958a49aec9c-kube-api-access-8bll8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.129728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.129359    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951dfe1a-96eb-4dde-b28d-7a951bdb48cd-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.131066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.130157    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:22.230969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.230367    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\" (UniqueName: \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\") pod \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") "
Oct 30 19:15:22.230969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.230837    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\") pod \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\" (UID: \"a754bed2-f5ce-4b96-96f2-3a38ba8b264b\") "
Oct 30 19:15:22.232515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.232490    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca" pod "a754bed2-f5ce-4b96-96f2-3a38ba8b264b" (UID: "a754bed2-f5ce-4b96-96f2-3a38ba8b264b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.238165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.238140    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6" pod "a754bed2-f5ce-4b96-96f2-3a38ba8b264b" (UID: "a754bed2-f5ce-4b96-96f2-3a38ba8b264b"). InnerVolumeSpecName "kube-api-access-sspl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.248534 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.248515    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:22.276114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.276077    5065 generic.go:356] "Generic (PLEG): container finished" podID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerID="c2f26694b7715f06dae68cb44a60de97239d60cfcf19c3a5a1f8a0ff3e94f7b1" exitCode=60
Oct 30 19:15:22.276226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.276155    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerDied","Data":"c2f26694b7715f06dae68cb44a60de97239d60cfcf19c3a5a1f8a0ff3e94f7b1"}
Oct 30 19:15:22.276226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.276205    5065 scope.go:122] "RemoveContainer" containerID="438456eb9e028f8d21a870ce0fdd44e440693869870188d16d09fb9006a4c25a"
Oct 30 19:15:22.276614 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.276596    5065 scope.go:122] "RemoveContainer" containerID="c2f26694b7715f06dae68cb44a60de97239d60cfcf19c3a5a1f8a0ff3e94f7b1"
Oct 30 19:15:22.277003 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:22.276978    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt_odh-ai-gateway-infra(8ba32f64-738c-4165-8b03-0a9fd816d9ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea"
Oct 30 19:15:22.278530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.278510    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerStarted","Data":"edb352fc611975391a212ec84b978615284c0bfd93d446aab4470ae11c74ced5"}
Oct 30 19:15:22.280851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.280828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerStarted","Data":"2438bc19d965b19fb49517fe066428d49f29517029767e784d7e86f6bf7718fc"}
Oct 30 19:15:22.284612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.284587    5065 generic.go:356] "Generic (PLEG): container finished" podID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerID="817fc912be731088b9ee773e7341e2c49c1fcaaba5d8c649b507a4a2a8d83584" exitCode=60
Oct 30 19:15:22.284725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.284706    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerDied","Data":"817fc912be731088b9ee773e7341e2c49c1fcaaba5d8c649b507a4a2a8d83584"}
Oct 30 19:15:22.293409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.293386    5065 generic.go:356] "Generic (PLEG): container finished" podID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerID="768075306cdf97eb67cb0dd383595f7101f0f525206464c8eceec1a14fe11e1f" exitCode=60
Oct 30 19:15:22.293553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.293494    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerDied","Data":"768075306cdf97eb67cb0dd383595f7101f0f525206464c8eceec1a14fe11e1f"}
Oct 30 19:15:22.293798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.293781    5065 scope.go:122] "RemoveContainer" containerID="768075306cdf97eb67cb0dd383595f7101f0f525206464c8eceec1a14fe11e1f"
Oct 30 19:15:22.294431 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:22.294402    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp_odh-ai-gateway-infra(f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d"
Oct 30 19:15:22.300428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.300381    5065 generic.go:356] "Generic (PLEG): container finished" podID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerID="f3fa8dcc010331287d8d181cb148147f6db4e6fc56b831397cadeed6cec6b4df" exitCode=60
Oct 30 19:15:22.300556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.300519    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerDied","Data":"f3fa8dcc010331287d8d181cb148147f6db4e6fc56b831397cadeed6cec6b4df"}
Oct 30 19:15:22.300954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.300927    5065 scope.go:122] "RemoveContainer" containerID="f4342aba41bc6eb57d904d5da35cc5c8b2e0cd9fcac608627cadeec29dcddef5"
Oct 30 19:15:22.301469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.301446    5065 scope.go:122] "RemoveContainer" containerID="f3fa8dcc010331287d8d181cb148147f6db4e6fc56b831397cadeed6cec6b4df"
Oct 30 19:15:22.302143 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:22.301934    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s_odh-ai-gateway-infra(4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e"
Oct 30 19:15:22.306940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.306915    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw" event={"ID":"6271310d-be4e-4a1d-a43e-f3b264fc9dea","Type":"ContainerDied","Data":"6a3ba8e11a8a8638085f4dad61978fd21842513e4efc965fa720614af02d603f"}
Oct 30 19:15:22.307060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.307010    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"
Oct 30 19:15:22.311982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.311236    5065 generic.go:356] "Generic (PLEG): container finished" podID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerID="0da54a0dd86c16aaac32147afdee8c94e70831d78ee68e4745a39069ddd7722d" exitCode=60
Oct 30 19:15:22.311982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.311361    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerDied","Data":"0da54a0dd86c16aaac32147afdee8c94e70831d78ee68e4745a39069ddd7722d"}
Oct 30 19:15:22.312284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.312111    5065 scope.go:122] "RemoveContainer" containerID="0da54a0dd86c16aaac32147afdee8c94e70831d78ee68e4745a39069ddd7722d"
Oct 30 19:15:22.312679 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:22.312632    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm_odh-ai-gateway-infra(6c229b7b-8d99-4441-a5d2-591584b9b780)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780"
Oct 30 19:15:22.314909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.314890    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x" event={"ID":"104c4ab4-75c5-413e-b338-850ec7f0b2d4","Type":"ContainerDied","Data":"ebb6eee15606e7e78124475185e4610e515a4557da57a343d10f48a3a4726b35"}
Oct 30 19:15:22.315138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.315122    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"
Oct 30 19:15:22.320735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.319999    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl" event={"ID":"a754bed2-f5ce-4b96-96f2-3a38ba8b264b","Type":"ContainerDied","Data":"10dad1ed149b9e58f012f303204409b886e53df582e895f781d322cb7aa4ef48"}
Oct 30 19:15:22.320735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.320079    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"
Oct 30 19:15:22.325725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.325660    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq" event={"ID":"219ac27f-79b4-4973-a403-4958a49aec9c","Type":"ContainerDied","Data":"542cec21abccb1b3b8bff88649ab43744b7f6b39e7dc95066f75c6de4009923e"}
Oct 30 19:15:22.325954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.325935    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"
Oct 30 19:15:22.328975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.328950    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4" event={"ID":"951dfe1a-96eb-4dde-b28d-7a951bdb48cd","Type":"ContainerDied","Data":"f7d3a0b9c491511638746ae6fb370861e5f258888d9752c7737391c20b6f0896"}
Oct 30 19:15:22.329073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.329034    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"
Oct 30 19:15:22.338632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.336406    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.338632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.336434    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspl6\" (UniqueName: \"kubernetes.io/projected/a754bed2-f5ce-4b96-96f2-3a38ba8b264b-kube-api-access-sspl6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.348447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.348332    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:22.348923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.348815    5065 scope.go:122] "RemoveContainer" containerID="8a4227c9a8cd84499a2fc31fba60339b60d63dd886fd1b0dac939bf2ac412d2e"
Oct 30 19:15:22.382259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.380793    5065 scope.go:122] "RemoveContainer" containerID="63dc2986b0a05b708e2fb60600d5fd7ea6ab793b2529e77c474ea7e2dee79ac6"
Oct 30 19:15:22.436412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.436327    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:22.536960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.536931    5065 scope.go:122] "RemoveContainer" containerID="12df46ce8f20fdea406c9bb24d2c9c682e4506f6c45612b955e813cc41e79d1f"
Oct 30 19:15:22.559140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.559097    5065 scope.go:122] "RemoveContainer" containerID="35f1b00d303641d5ffb784bf28887b9308d7b3bab25b05c90a1f14668f1068ba"
Oct 30 19:15:22.560294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.560240    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" podStartSLOduration=22.560222796 podStartE2EDuration="22.560222796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.544298192 +0000 UTC m=+2542.807763142" watchObservedRunningTime="2026-10-30 19:15:22.560222796 +0000 UTC m=+2542.823687745"
Oct 30 19:15:22.575398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.575315    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:22.589036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.589014    5065 scope.go:122] "RemoveContainer" containerID="633279bc624cfe852bfd683a6c737d5e37c951c4c02d5b4cd21c87b0a8ec8880"
Oct 30 19:15:22.614341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.614317    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:22.637744 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.637717    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:22.643279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.643257    5065 scope.go:122] "RemoveContainer" containerID="8abe30072bc560f2ba55efd84825b8a9e7500266b8f5d26e9eb5186f28801be6"
Oct 30 19:15:22.643913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.643883    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\") pod \"015c3c4f-3808-4291-b4c3-9b8ab0453512\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") "
Oct 30 19:15:22.644028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.643990    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\" (UniqueName: \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\") pod \"015c3c4f-3808-4291-b4c3-9b8ab0453512\" (UID: \"015c3c4f-3808-4291-b4c3-9b8ab0453512\") "
Oct 30 19:15:22.648201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.648068    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn" pod "015c3c4f-3808-4291-b4c3-9b8ab0453512" (UID: "015c3c4f-3808-4291-b4c3-9b8ab0453512"). InnerVolumeSpecName "kube-api-access-p6lfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.649735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.648428    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca" pod "015c3c4f-3808-4291-b4c3-9b8ab0453512" (UID: "015c3c4f-3808-4291-b4c3-9b8ab0453512"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.657103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.657068    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:22.661068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.661041    5065 scope.go:122] "RemoveContainer" containerID="d2fd411f3d0ef78c8688f98cb2ae56acfadebb17689b35dca28198ed1531e4b7"
Oct 30 19:15:22.672360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.672345    5065 scope.go:122] "RemoveContainer" containerID="6107456683f7f20138723b32031f24479857f82b81327b06e10893a99ef52d26"
Oct 30 19:15:22.712931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.712900    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:22.718766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.718742    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:22.731358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.731336    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c6027-29889795-xvkvl"]
Oct 30 19:15:22.746513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746486    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\") pod \"486a9406-1e09-4f18-8efd-63e6db68afb6\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") "
Oct 30 19:15:22.746647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746523    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\" (UniqueName: \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\") pod \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") "
Oct 30 19:15:22.746647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746562    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\") pod \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\" (UID: \"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4\") "
Oct 30 19:15:22.746779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746693    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\" (UniqueName: \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\") pod \"486a9406-1e09-4f18-8efd-63e6db68afb6\" (UID: \"486a9406-1e09-4f18-8efd-63e6db68afb6\") "
Oct 30 19:15:22.746929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746899    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca" pod "486a9406-1e09-4f18-8efd-63e6db68afb6" (UID: "486a9406-1e09-4f18-8efd-63e6db68afb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.747036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.746931    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca" pod "bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" (UID: "bc4e2793-5413-40bf-ac7f-6f63c9ce48c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.747418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.747404    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a9406-1e09-4f18-8efd-63e6db68afb6-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.747475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.747421    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.747475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.747432    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/015c3c4f-3808-4291-b4c3-9b8ab0453512-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.747475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.747441    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6lfn\" (UniqueName: \"kubernetes.io/projected/015c3c4f-3808-4291-b4c3-9b8ab0453512-kube-api-access-p6lfn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.750057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.750018    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s" pod "486a9406-1e09-4f18-8efd-63e6db68afb6" (UID: "486a9406-1e09-4f18-8efd-63e6db68afb6"). InnerVolumeSpecName "kube-api-access-gb28s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.767125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.767066    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n" pod "bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" (UID: "bc4e2793-5413-40bf-ac7f-6f63c9ce48c4"). InnerVolumeSpecName "kube-api-access-xvw8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.771558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.771530    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:22.785157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.782801    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889795-vtr6x"]
Oct 30 19:15:22.788210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.788185    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:22.793702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.793679    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889795-kddz4"]
Oct 30 19:15:22.849283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.849212    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw8n\" (UniqueName: \"kubernetes.io/projected/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4-kube-api-access-xvw8n\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.849283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.849239    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb28s\" (UniqueName: \"kubernetes.io/projected/486a9406-1e09-4f18-8efd-63e6db68afb6-kube-api-access-gb28s\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.858356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.858331    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:22.866347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.866322    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e1e1ff-29889795-k8ptw"]
Oct 30 19:15:22.891827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.891805    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:22.902068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:22.902040    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-784fbf-29889795-zs6mq"]
Oct 30 19:15:23.091824 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:23.091785    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:15:23.340396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.340366    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"
Oct 30 19:15:23.340572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.340391    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g" event={"ID":"bc4e2793-5413-40bf-ac7f-6f63c9ce48c4","Type":"ContainerDied","Data":"373efceabf15a7df337e7c652151a3696ad47e4d097eeff3d367e38a7a108ad5"}
Oct 30 19:15:23.340572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.340459    5065 scope.go:122] "RemoveContainer" containerID="817fc912be731088b9ee773e7341e2c49c1fcaaba5d8c649b507a4a2a8d83584"
Oct 30 19:15:23.342298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.342263    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q" event={"ID":"015c3c4f-3808-4291-b4c3-9b8ab0453512","Type":"ContainerDied","Data":"1b89d364a6a8072090b128f3f191a7e5da8ab2a2028a42865267255562aa4d7c"}
Oct 30 19:15:23.342441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.342271    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"
Oct 30 19:15:23.364765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.363628    5065 scope.go:122] "RemoveContainer" containerID="580cf37cd99ee8c4000018ebf783cb81db677df1aabf44baa07664df35c14da2"
Oct 30 19:15:23.367746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.365289    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs" event={"ID":"486a9406-1e09-4f18-8efd-63e6db68afb6","Type":"ContainerDied","Data":"441a349984104b66bf9265e49f8505e04548c8a2363e00f0ac746925621e49c0"}
Oct 30 19:15:23.367746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.365566    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:23.367746 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.365321    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"
Oct 30 19:15:23.385206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.384784    5065 generic.go:356] "Generic (PLEG): container finished" podID="0107a981-57e3-438d-96ef-bb845a202804" containerID="edb352fc611975391a212ec84b978615284c0bfd93d446aab4470ae11c74ced5" exitCode=60
Oct 30 19:15:23.385206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.384886    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerDied","Data":"edb352fc611975391a212ec84b978615284c0bfd93d446aab4470ae11c74ced5"}
Oct 30 19:15:23.399169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.396417    5065 generic.go:356] "Generic (PLEG): container finished" podID="28fad370-0534-487f-aa08-d04982ebc22e" containerID="2438bc19d965b19fb49517fe066428d49f29517029767e784d7e86f6bf7718fc" exitCode=60
Oct 30 19:15:23.399169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.396466    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerDied","Data":"2438bc19d965b19fb49517fe066428d49f29517029767e784d7e86f6bf7718fc"}
Oct 30 19:15:23.399169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.396959    5065 scope.go:122] "RemoveContainer" containerID="2438bc19d965b19fb49517fe066428d49f29517029767e784d7e86f6bf7718fc"
Oct 30 19:15:23.399169 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:23.397490    5065 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q_odh-ai-gateway-infra(28fad370-0534-487f-aa08-d04982ebc22e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" podUID="28fad370-0534-487f-aa08-d04982ebc22e"
Oct 30 19:15:23.417302 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.417272    5065 scope.go:122] "RemoveContainer" containerID="907b859958bc484ffd1595ae192db0e22b19ab64e3c12b1e19ceced79191900a"
Oct 30 19:15:23.494131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.493989    5065 scope.go:122] "RemoveContainer" containerID="f5a2ae4fc8199cd99212361d5a59c6c993cc2b3191b8bcad7a55c3ea9ee350c3"
Oct 30 19:15:23.523191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.523167    5065 scope.go:122] "RemoveContainer" containerID="8d55d6ac127a1756f3155eb2071c2d95d38b8e34a47b22dbaa6cd6c43422fe50"
Oct 30 19:15:23.550648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.550377    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:23.560813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.560433    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4e94aa-29889795-2gj5g"]
Oct 30 19:15:23.566796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.566769    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:23.586285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.586261    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:23.601520 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.601462    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-de4bda-29889795-vk55q"]
Oct 30 19:15:23.662871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.662848    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:23.668064 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.667986    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:23.674213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.674194    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d72beb-29889795-dvwrs"]
Oct 30 19:15:23.704829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.702445    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\") pod \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") "
Oct 30 19:15:23.704829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.702582    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\" (UniqueName: \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\") pod \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\" (UID: \"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d\") "
Oct 30 19:15:23.704829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.704750    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca" pod "f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" (UID: "f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.705898 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.705859    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.716841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.716786    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k" pod "f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" (UID: "f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d"). InnerVolumeSpecName "kube-api-access-j6c5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.796475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.796443    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:23.801523 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.801480    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:23.806841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.806810    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\") pod \"6c229b7b-8d99-4441-a5d2-591584b9b780\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") "
Oct 30 19:15:23.806990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.806962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\" (UniqueName: \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\") pod \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") "
Oct 30 19:15:23.807313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.807288    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\") pod \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\" (UID: \"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e\") "
Oct 30 19:15:23.807424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.807335    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\") pod \"6c229b7b-8d99-4441-a5d2-591584b9b780\" (UID: \"6c229b7b-8d99-4441-a5d2-591584b9b780\") "
Oct 30 19:15:23.808183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.808137    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca" pod "6c229b7b-8d99-4441-a5d2-591584b9b780" (UID: "6c229b7b-8d99-4441-a5d2-591584b9b780"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.808663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.808643    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca" pod "4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" (UID: "4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.808797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.808700    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6c5k\" (UniqueName: \"kubernetes.io/projected/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d-kube-api-access-j6c5k\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.808914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.808808    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c229b7b-8d99-4441-a5d2-591584b9b780-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.810527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.810492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft" pod "4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" (UID: "4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e"). InnerVolumeSpecName "kube-api-access-s55ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.812170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.811462    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7" pod "6c229b7b-8d99-4441-a5d2-591584b9b780" (UID: "6c229b7b-8d99-4441-a5d2-591584b9b780"). InnerVolumeSpecName "kube-api-access-pkjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.847198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.847165    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:23.876609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.872551    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:23.901223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.901181    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" path="/var/lib/kubelet/pods/015c3c4f-3808-4291-b4c3-9b8ab0453512/volumes"
Oct 30 19:15:23.901945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.901920    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" path="/var/lib/kubelet/pods/104c4ab4-75c5-413e-b338-850ec7f0b2d4/volumes"
Oct 30 19:15:23.902587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.902535    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" path="/var/lib/kubelet/pods/219ac27f-79b4-4973-a403-4958a49aec9c/volumes"
Oct 30 19:15:23.904290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.904264    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" path="/var/lib/kubelet/pods/486a9406-1e09-4f18-8efd-63e6db68afb6/volumes"
Oct 30 19:15:23.904874 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.904853    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" path="/var/lib/kubelet/pods/6271310d-be4e-4a1d-a43e-f3b264fc9dea/volumes"
Oct 30 19:15:23.905418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.905397    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" path="/var/lib/kubelet/pods/951dfe1a-96eb-4dde-b28d-7a951bdb48cd/volumes"
Oct 30 19:15:23.906069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.906050    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" path="/var/lib/kubelet/pods/a754bed2-f5ce-4b96-96f2-3a38ba8b264b/volumes"
Oct 30 19:15:23.906637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.906615    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" path="/var/lib/kubelet/pods/bc4e2793-5413-40bf-ac7f-6f63c9ce48c4/volumes"
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.909344    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\" (UniqueName: \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\") pod \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") "
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.909406    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\") pod \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\" (UID: \"8ba32f64-738c-4165-8b03-0a9fd816d9ea\") "
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.909558    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\" (UniqueName: \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\") pod \"0107a981-57e3-438d-96ef-bb845a202804\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") "
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.909605    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\") pod \"0107a981-57e3-438d-96ef-bb845a202804\" (UID: \"0107a981-57e3-438d-96ef-bb845a202804\") "
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.910178    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.910193    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjv7\" (UniqueName: \"kubernetes.io/projected/6c229b7b-8d99-4441-a5d2-591584b9b780-kube-api-access-pkjv7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.910230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.910204    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s55ft\" (UniqueName: \"kubernetes.io/projected/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e-kube-api-access-s55ft\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.910665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.910442    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca" pod "0107a981-57e3-438d-96ef-bb845a202804" (UID: "0107a981-57e3-438d-96ef-bb845a202804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.912155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.912071    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca" pod "8ba32f64-738c-4165-8b03-0a9fd816d9ea" (UID: "8ba32f64-738c-4165-8b03-0a9fd816d9ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.914503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.913402    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr" pod "8ba32f64-738c-4165-8b03-0a9fd816d9ea" (UID: "8ba32f64-738c-4165-8b03-0a9fd816d9ea"). InnerVolumeSpecName "kube-api-access-xg5hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.930250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:23.930216    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf" pod "0107a981-57e3-438d-96ef-bb845a202804" (UID: "0107a981-57e3-438d-96ef-bb845a202804"). InnerVolumeSpecName "kube-api-access-vc2lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:24.011629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.011594    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5hr\" (UniqueName: \"kubernetes.io/projected/8ba32f64-738c-4165-8b03-0a9fd816d9ea-kube-api-access-xg5hr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.011629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.011629    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba32f64-738c-4165-8b03-0a9fd816d9ea-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.011803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.011647    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc2lf\" (UniqueName: \"kubernetes.io/projected/0107a981-57e3-438d-96ef-bb845a202804-kube-api-access-vc2lf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.011803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.011670    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0107a981-57e3-438d-96ef-bb845a202804-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.448358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.448322    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm" event={"ID":"6c229b7b-8d99-4441-a5d2-591584b9b780","Type":"ContainerDied","Data":"e3e2c9c22cdd262ed59aade27afb3896131c2121d5add1458981f53def2b1caa"}
Oct 30 19:15:24.449206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.449188    5065 scope.go:122] "RemoveContainer" containerID="0da54a0dd86c16aaac32147afdee8c94e70831d78ee68e4745a39069ddd7722d"
Oct 30 19:15:24.450689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.449050    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"
Oct 30 19:15:24.453258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.453230    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt" event={"ID":"8ba32f64-738c-4165-8b03-0a9fd816d9ea","Type":"ContainerDied","Data":"6447c2e03d26c81f7f54e74858d54209fd59d0326f99b71fc298b5334fa1514e"}
Oct 30 19:15:24.453368 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.453326    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"
Oct 30 19:15:24.457267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.455917    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94" event={"ID":"0107a981-57e3-438d-96ef-bb845a202804","Type":"ContainerDied","Data":"9fbf6c18d92fa55f35fe5f541f1446b0e2e2568450d9fff815df2a7f5bd7c573"}
Oct 30 19:15:24.457267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.456006    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"
Oct 30 19:15:24.472888 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.472866    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"
Oct 30 19:15:24.473011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.472890    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp" event={"ID":"f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d","Type":"ContainerDied","Data":"f8560acf23eaf5724a729012374efdfdf929102a6c35c44d5e8445155327b174"}
Oct 30 19:15:24.474789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.474671    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s" event={"ID":"4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e","Type":"ContainerDied","Data":"d9794f2b7d309a346b5ecbef7cb0234202539d1696e1587bc5c6e52ac0cdb984"}
Oct 30 19:15:24.474959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.474935    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"
Oct 30 19:15:24.481824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.481805    5065 scope.go:122] "RemoveContainer" containerID="c2f26694b7715f06dae68cb44a60de97239d60cfcf19c3a5a1f8a0ff3e94f7b1"
Oct 30 19:15:24.494732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.494710    5065 scope.go:122] "RemoveContainer" containerID="edb352fc611975391a212ec84b978615284c0bfd93d446aab4470ae11c74ced5"
Oct 30 19:15:24.513461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.513423    5065 scope.go:122] "RemoveContainer" containerID="768075306cdf97eb67cb0dd383595f7101f0f525206464c8eceec1a14fe11e1f"
Oct 30 19:15:24.525151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.523363    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:24.530868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.530844    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a717c7-29889795-lnckm"]
Oct 30 19:15:24.531318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.531298    5065 scope.go:122] "RemoveContainer" containerID="f3fa8dcc010331287d8d181cb148147f6db4e6fc56b831397cadeed6cec6b4df"
Oct 30 19:15:24.591977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.591946    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:24.601326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.601279    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b0e009-29889795-5ts94"]
Oct 30 19:15:24.632104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.632062    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:24.639263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.639234    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f7c6e3-29889795-729rp"]
Oct 30 19:15:24.645197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.645171    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:24.652351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.652331    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a962a-29889795-2s66s"]
Oct 30 19:15:24.674481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.674456    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:24.683063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.683040    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-84c32a-29889795-gw4qt"]
Oct 30 19:15:24.697367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.697347    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:24.728926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.728874    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\") pod \"28fad370-0534-487f-aa08-d04982ebc22e\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") "
Oct 30 19:15:24.729031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.729020    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\" (UniqueName: \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\") pod \"28fad370-0534-487f-aa08-d04982ebc22e\" (UID: \"28fad370-0534-487f-aa08-d04982ebc22e\") "
Oct 30 19:15:24.729348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.729318    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca" pod "28fad370-0534-487f-aa08-d04982ebc22e" (UID: "28fad370-0534-487f-aa08-d04982ebc22e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:24.730958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.730933    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fad370-0534-487f-aa08-d04982ebc22e-maas-api-service-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.738290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.738261    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr" pod "28fad370-0534-487f-aa08-d04982ebc22e" (UID: "28fad370-0534-487f-aa08-d04982ebc22e"). InnerVolumeSpecName "kube-api-access-7ttzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:24.832570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:24.832536    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ttzr\" (UniqueName: \"kubernetes.io/projected/28fad370-0534-487f-aa08-d04982ebc22e-kube-api-access-7ttzr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:25.485701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.485670    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"
Oct 30 19:15:25.486179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.485699    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q" event={"ID":"28fad370-0534-487f-aa08-d04982ebc22e","Type":"ContainerDied","Data":"2651e80a2ab87f457472c4aebd16bb87c336d2db28eebbe9061e2a834a619e24"}
Oct 30 19:15:25.486179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.485758    5065 scope.go:122] "RemoveContainer" containerID="2438bc19d965b19fb49517fe066428d49f29517029767e784d7e86f6bf7718fc"
Oct 30 19:15:25.628116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.617152    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:25.649222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.649182    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6d7870-29889795-mpl7q"]
Oct 30 19:15:25.900780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.900732    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0107a981-57e3-438d-96ef-bb845a202804" path="/var/lib/kubelet/pods/0107a981-57e3-438d-96ef-bb845a202804/volumes"
Oct 30 19:15:25.901571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.901541    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fad370-0534-487f-aa08-d04982ebc22e" path="/var/lib/kubelet/pods/28fad370-0534-487f-aa08-d04982ebc22e/volumes"
Oct 30 19:15:25.902329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.902306    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" path="/var/lib/kubelet/pods/4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e/volumes"
Oct 30 19:15:25.903466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.903441    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" path="/var/lib/kubelet/pods/6c229b7b-8d99-4441-a5d2-591584b9b780/volumes"
Oct 30 19:15:25.904207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.904185    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" path="/var/lib/kubelet/pods/8ba32f64-738c-4165-8b03-0a9fd816d9ea/volumes"
Oct 30 19:15:25.904918 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:25.904894    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" path="/var/lib/kubelet/pods/f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d/volumes"
Oct 30 19:15:28.031029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:28.030994    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"]
Oct 30 19:15:28.036530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:28.036503    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c5deb4-kzfwx"]
Oct 30 19:15:29.896136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:29.896080    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd92e381-0bab-4637-a96d-7f1c9bf13935" path="/var/lib/kubelet/pods/cd92e381-0bab-4637-a96d-7f1c9bf13935/volumes"
Oct 30 19:15:33.086510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:33.086476    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"]
Oct 30 19:15:33.092196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:33.092171    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71212b3c-mnzfj"]
Oct 30 19:15:33.898312 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:33.898276    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d684e4d-b989-4b39-b532-d5ac310147d8" path="/var/lib/kubelet/pods/5d684e4d-b989-4b39-b532-d5ac310147d8/volumes"
Oct 30 19:15:51.046714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:51.046635    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"]
Oct 30 19:15:51.057408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:51.057383    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a333bd-j5q64"]
Oct 30 19:15:51.898210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:15:51.898176    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0f7e03-7145-4880-b0c4-a6da2909a5e1" path="/var/lib/kubelet/pods/1d0f7e03-7145-4880-b0c4-a6da2909a5e1/volumes"
Oct 30 19:15:53.058046 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:15:53.058004    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:16:03.055948 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:03.055902    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"]
Oct 30 19:16:03.070459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:03.070379    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e304119-w6v8g"]
Oct 30 19:16:03.894411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:03.894369    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5" path="/var/lib/kubelet/pods/ea2e1e73-2a6b-44a1-a538-a5ef8f6517b5/volumes"
Oct 30 19:16:14.676288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:14.676260    5065 scope.go:122] "RemoveContainer" containerID="b966427aad779f7069914ca24b1ffb172de66e99cf6349dc8c525ef60d69e73b"
Oct 30 19:16:14.691150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:14.691125    5065 scope.go:122] "RemoveContainer" containerID="e11001d4d6df8925ec7099af3edfb04e48834d77f3725c94905ce85690726fe0"
Oct 30 19:16:15.034357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:15.034323    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"]
Oct 30 19:16:15.042210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:15.042171    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8290c0ce-qqqcz"]
Oct 30 19:16:15.903320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:15.903287    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b20f510-4181-418c-a499-4d1b061f9b43" path="/var/lib/kubelet/pods/7b20f510-4181-418c-a499-4d1b061f9b43/volumes"
Oct 30 19:16:16.042337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:16.042300    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"]
Oct 30 19:16:16.049201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:16.049168    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-592b8461-rpns7"]
Oct 30 19:16:17.893201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:17.893170    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5920988-b0bf-4441-8ac6-f2e9a205e6dd" path="/var/lib/kubelet/pods/d5920988-b0bf-4441-8ac6-f2e9a205e6dd/volumes"
Oct 30 19:16:22.043904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:22.043866    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"]
Oct 30 19:16:22.049318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:22.049290    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c159b54e-cm5zt"]
Oct 30 19:16:23.059180 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:16:23.059142    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:16:23.895800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:23.895766    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a679010-ea3d-437c-96c4-e79f57cadecc" path="/var/lib/kubelet/pods/2a679010-ea3d-437c-96c4-e79f57cadecc/volumes"
Oct 30 19:16:29.034977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:29.034943    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"]
Oct 30 19:16:29.040370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:29.040342    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89265e2-z56qp"]
Oct 30 19:16:29.894029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:29.893994    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff" path="/var/lib/kubelet/pods/25cf3c3e-4db1-4bec-9088-7ae6cdc3b5ff/volumes"
Oct 30 19:16:41.058691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:41.058654    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"]
Oct 30 19:16:41.064010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:41.063979    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46f68a25-88227"]
Oct 30 19:16:41.895888 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:41.895847    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7e09ea-3065-4d25-be85-8960f1075956" path="/var/lib/kubelet/pods/8c7e09ea-3065-4d25-be85-8960f1075956/volumes"
Oct 30 19:16:53.050961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:53.050928    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"]
Oct 30 19:16:53.056720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:53.056693    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36226956-kk7tq"]
Oct 30 19:16:53.061155 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:16:53.061130    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:16:53.898116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:16:53.897805    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a951824-d0d9-49f7-a574-ee08f231ed28" path="/var/lib/kubelet/pods/6a951824-d0d9-49f7-a574-ee08f231ed28/volumes"
Oct 30 19:17:05.266554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.266517    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"]
Oct 30 19:17:05.267488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267470    5065 cpu_manager.go:414] "containerMap: removing container" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267489    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267504    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267509    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267520    5065 cpu_manager.go:414] "containerMap: removing container" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267525    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267531    5065 cpu_manager.go:414] "containerMap: removing container" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267536    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267546    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267551    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267557    5065 cpu_manager.go:414] "containerMap: removing container" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267562    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267568    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.267570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267572    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267578    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267582    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267592    5065 cpu_manager.go:414] "containerMap: removing container" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267596    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267602    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267606    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267612    5065 cpu_manager.go:414] "containerMap: removing container" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267616    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267622    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267626    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267637    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267641    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267646    5065 cpu_manager.go:414] "containerMap: removing container" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267650    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267658    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267662    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267676    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267681    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267688    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267692    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267697    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267701    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267709    5065 cpu_manager.go:414] "containerMap: removing container" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267713    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267720    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267724    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267734    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267738    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267743    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267748    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267755    5065 cpu_manager.go:414] "containerMap: removing container" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267760    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267766    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267769    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267775    5065 cpu_manager.go:414] "containerMap: removing container" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267779    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267788    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267793    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267799    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267803    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267809    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267813    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267822    5065 cpu_manager.go:414] "containerMap: removing container" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267826    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267838    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267842    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267854    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267858    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267865    5065 cpu_manager.go:414] "containerMap: removing container" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267869    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267876    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267880    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267885    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267889    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267902    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267906    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267912    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267916    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267924    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267927    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267934    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267937    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267948    5065 cpu_manager.go:414] "containerMap: removing container" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.268007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.267953    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268079    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268105    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268115    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268121    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268131    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268137    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268141    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268147    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268154    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268158    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268162    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268170    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268175    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268180    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268184    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268190    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268195    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268200    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268205    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268211    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268217    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268222    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec7cab6-53dc-406b-9ae6-81386cc82a51" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268227    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268231    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268237    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268244    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268251    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268282    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268287    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be31543-8f9c-47ca-b794-a6000967be6a" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268295    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268303    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4e2793-5413-40bf-ac7f-6f63c9ce48c4" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268310    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268317    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268325    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268330    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268336    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268345    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268350    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268356    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:05.270132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.268360    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:05.272593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.272577    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.277612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.277589    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:05.279357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.279336    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:17:05.281454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.281242    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"]
Oct 30 19:17:05.380668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.380624    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6gp\" (UniqueName: \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.380880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.380856    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.380946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.380922    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.482662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.482623    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6gp\" (UniqueName: \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.482862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.482790    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.482862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.482837    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.483771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.483739    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.485449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.485425    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.497199 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.497170    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6gp\" (UniqueName: \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\") pod \"maas-api-revoke-sub-673606ee-kxrpd\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.591142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.591061    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:05.899572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:05.899124    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"]
Oct 30 19:17:06.068506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:06.068473    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd" event={"ID":"2e86642d-7fc4-4171-bb7f-0221ea30334e","Type":"ContainerStarted","Data":"6c25429e800eb329aa6bf71b756081a6acfa08d3ee7033f01eb515f3aad3b81c"}
Oct 30 19:17:07.099016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:07.098973    5065 generic.go:356] "Generic (PLEG): container finished" podID="2e86642d-7fc4-4171-bb7f-0221ea30334e" containerID="61858fa3e539a964de4c763a9d70f37432f22518554735a672be355dac48154b" exitCode=0
Oct 30 19:17:07.099615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:07.099322    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd" event={"ID":"2e86642d-7fc4-4171-bb7f-0221ea30334e","Type":"ContainerDied","Data":"61858fa3e539a964de4c763a9d70f37432f22518554735a672be355dac48154b"}
Oct 30 19:17:08.153780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.153738    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:17:08.154187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.153996    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" containerID="cri-o://575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d" gracePeriod=30
Oct 30 19:17:08.409003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.408977    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:08.559447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.559416    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\" (UniqueName: \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\") pod \"2e86642d-7fc4-4171-bb7f-0221ea30334e\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") "
Oct 30 19:17:08.559621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.559499    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\") pod \"2e86642d-7fc4-4171-bb7f-0221ea30334e\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") "
Oct 30 19:17:08.559621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.559599    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\") pod \"2e86642d-7fc4-4171-bb7f-0221ea30334e\" (UID: \"2e86642d-7fc4-4171-bb7f-0221ea30334e\") "
Oct 30 19:17:08.559864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.559839    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca" pod "2e86642d-7fc4-4171-bb7f-0221ea30334e" (UID: "2e86642d-7fc4-4171-bb7f-0221ea30334e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:08.560209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.560192    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.561753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.561729    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca" pod "2e86642d-7fc4-4171-bb7f-0221ea30334e" (UID: "2e86642d-7fc4-4171-bb7f-0221ea30334e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:08.561881 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.561862    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp" pod "2e86642d-7fc4-4171-bb7f-0221ea30334e" (UID: "2e86642d-7fc4-4171-bb7f-0221ea30334e"). InnerVolumeSpecName "kube-api-access-fv6gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:08.661449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.661388    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6gp\" (UniqueName: \"kubernetes.io/projected/2e86642d-7fc4-4171-bb7f-0221ea30334e-kube-api-access-fv6gp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.661449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:08.661415    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e86642d-7fc4-4171-bb7f-0221ea30334e-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.120427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:09.120395    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"
Oct 30 19:17:09.120616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:09.120427    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd" event={"ID":"2e86642d-7fc4-4171-bb7f-0221ea30334e","Type":"ContainerDied","Data":"6c25429e800eb329aa6bf71b756081a6acfa08d3ee7033f01eb515f3aad3b81c"}
Oct 30 19:17:09.120616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:09.120484    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c25429e800eb329aa6bf71b756081a6acfa08d3ee7033f01eb515f3aad3b81c"
Oct 30 19:17:11.370688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:11.370602    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:17:11.370688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:11.370654    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:17:13.578071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.578038    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:17:13.760280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760243    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760335    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760366    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760407    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760611    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760636    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.760799 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760670    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.760861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760795    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.760861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760847    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.760984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760870    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.760984 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760906    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.761125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.760998    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.761306 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.761220    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\") pod \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\" (UID: \"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4\") "
Oct 30 19:17:13.761408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.761330    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:13.762158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.762129    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.762283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.762161    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.762283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.762178    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.762283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.762194    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.762283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.762210    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.763562 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.763535    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:13.764105 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.764066    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.764317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.764290    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:13.764404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.764327    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt" pod "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" (UID: "7e7bb8ad-aa79-430e-91e5-97fd8829c2e4"). InnerVolumeSpecName "kube-api-access-s7fjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:13.863426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.863360    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fjt\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-kube-api-access-s7fjt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.863426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.863385    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.863426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.863395    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.863426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:13.863403    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.183769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.183652    5065 generic.go:356] "Generic (PLEG): container finished" podID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerID="575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d" exitCode=0
Oct 30 19:17:14.184053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.184015    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerDied","Data":"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"}
Oct 30 19:17:14.184236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.184100    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv" event={"ID":"7e7bb8ad-aa79-430e-91e5-97fd8829c2e4","Type":"ContainerDied","Data":"cbbbc8e768033cc6d4b4c6a314683d694d8190549a5b458d84c9c5779ae0d3c2"}
Oct 30 19:17:14.184236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.184134    5065 scope.go:122] "RemoveContainer" containerID="575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"
Oct 30 19:17:14.184410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.184322    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"
Oct 30 19:17:14.202424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.202404    5065 scope.go:122] "RemoveContainer" containerID="b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"
Oct 30 19:17:14.217674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.217647    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:17:14.228404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.228197    5065 scope.go:122] "RemoveContainer" containerID="575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"
Oct 30 19:17:14.230178 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:14.230056    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d\": container with ID starting with 575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d not found: ID does not exist" containerID="575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"
Oct 30 19:17:14.230358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.230323    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d"} err="failed to get container status \"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d\": rpc error: code = NotFound desc = could not find container \"575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d\": container with ID starting with 575fcdc4a1b1c857e06b83f29085556232c87264762ba0974b3bb049d29f812d not found: ID does not exist"
Oct 30 19:17:14.230482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.230362    5065 scope.go:122] "RemoveContainer" containerID="b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"
Oct 30 19:17:14.230693 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:14.230670    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9\": container with ID starting with b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9 not found: ID does not exist" containerID="b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"
Oct 30 19:17:14.230766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.230699    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9"} err="failed to get container status \"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9\": rpc error: code = NotFound desc = could not find container \"b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9\": container with ID starting with b0457794c51dee7613d4c3a3eac193b44383571ad15e80c3133005ad73ad8ef9 not found: ID does not exist"
Oct 30 19:17:14.233492 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.233471    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-ggxlv"]
Oct 30 19:17:14.893929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:14.893893    5065 scope.go:122] "RemoveContainer" containerID="ede8e73bf29c290dc52f3c28549c33f112d62761cda4b77d60b37f5231df6361"
Oct 30 19:17:15.023387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.019736    5065 scope.go:122] "RemoveContainer" containerID="272bff93d830b3fee5365eea5707b69151dff15c2143d7ce1661d66a0aaf2261"
Oct 30 19:17:15.035702 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.035654    5065 scope.go:122] "RemoveContainer" containerID="234e980c0dc28b90f1c6d54256c08a60a7f45050cca783234c0f4bebfc1a4b70"
Oct 30 19:17:15.049825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.049800    5065 scope.go:122] "RemoveContainer" containerID="ed10509787e5b70f6699f595223e10d1fa46d455a4e4953c86077582b8915fcc"
Oct 30 19:17:15.063619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.063597    5065 scope.go:122] "RemoveContainer" containerID="3fa56b9727e27485b9c11d8065856bb82e15c42bd85833a7e53ca2373b3bbc6d"
Oct 30 19:17:15.075137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.075111    5065 scope.go:122] "RemoveContainer" containerID="c2a2f857cc3a20153d205c91ed4cefbde5e7240793683408a1a43664daf924d8"
Oct 30 19:17:15.894175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:15.894145    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" path="/var/lib/kubelet/pods/7e7bb8ad-aa79-430e-91e5-97fd8829c2e4/volumes"
Oct 30 19:17:16.964844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.964813    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"]
Oct 30 19:17:16.965771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965743    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:16.965771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965764    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965796    5065 cpu_manager.go:414] "containerMap: removing container" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965802    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965811    5065 cpu_manager.go:414] "containerMap: removing container" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965816    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965826    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965830    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965836    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2e86642d-7fc4-4171-bb7f-0221ea30334e" containerName="revoke-keys"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965840    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e86642d-7fc4-4171-bb7f-0221ea30334e" containerName="revoke-keys"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965848    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965855    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a754bed2-f5ce-4b96-96f2-3a38ba8b264b" containerName="cleanup"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965872    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.965884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965877    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965894    5065 cpu_manager.go:414] "containerMap: removing container" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965898    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965911    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965915    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021550d-953a-41bb-b1ec-10fec61689f3" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965942    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965949    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965962    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965966    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0107a981-57e3-438d-96ef-bb845a202804" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965979    5065 cpu_manager.go:414] "containerMap: removing container" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965983    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965991    5065 cpu_manager.go:414] "containerMap: removing container" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.965995    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966004    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966008    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966023    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966031    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966186    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6271310d-be4e-4a1d-a43e-f3b264fc9dea" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966199    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="486a9406-1e09-4f18-8efd-63e6db68afb6" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966212    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966220    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e86642d-7fc4-4171-bb7f-0221ea30334e" containerName="revoke-keys"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966227    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="951dfe1a-96eb-4dde-b28d-7a951bdb48cd" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966234    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7bb8ad-aa79-430e-91e5-97fd8829c2e4" containerName="istio-proxy"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966239    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7baa188-90a2-4a8b-b5fc-2de07b5c7f6d" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966248    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fad370-0534-487f-aa08-d04982ebc22e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966257    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="104c4ab4-75c5-413e-b338-850ec7f0b2d4" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966264    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c3c4f-3808-4291-b4c3-9b8ab0453512" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966269    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b74cc2c-7747-4bb4-8830-7d1f8f67fe9e" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966275    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba32f64-738c-4165-8b03-0a9fd816d9ea" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966282    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c229b7b-8d99-4441-a5d2-591584b9b780" containerName="cleanup"
Oct 30 19:17:16.966335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.966294    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="219ac27f-79b4-4973-a403-4958a49aec9c" containerName="cleanup"
Oct 30 19:17:16.974011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.972748    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:16.982590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.981265    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"]
Oct 30 19:17:16.982590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.981540    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:16.982590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.981636    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:17:16.998472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.998447    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w74\" (UniqueName: \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:16.998602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.998504    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:16.998658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:16.998638    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.100386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.100347    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w74\" (UniqueName: \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.100527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.100436    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.100590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.100576    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.101183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.101155    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.103215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.103192    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.124159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.124137    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w74\" (UniqueName: \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\") pod \"maas-api-revoke-sub-0703002e-fzpx6\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.292946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.292916    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:17.640044 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:17:17.640002    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd97c7a_da68_4d65_8804_a0c13ab96e37.slice/crio-21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a WatchSource:0}: Error finding container 21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a: Status 404 returned error can't find the container with id 21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a
Oct 30 19:17:17.640931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:17.640900    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"]
Oct 30 19:17:18.278600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:18.278566    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6" event={"ID":"ebd97c7a-da68-4d65-8804-a0c13ab96e37","Type":"ContainerStarted","Data":"adec58811b6f325c32039c84b53b008cb7e0afc23e49044bcf36f92c4a861619"}
Oct 30 19:17:18.278983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:18.278620    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6" event={"ID":"ebd97c7a-da68-4d65-8804-a0c13ab96e37","Type":"ContainerStarted","Data":"21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a"}
Oct 30 19:17:18.301698 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:18.301641    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6" podStartSLOduration=2.301622734 podStartE2EDuration="2.301622734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:18.298666015 +0000 UTC m=+2658.562130965" watchObservedRunningTime="2026-10-30 19:17:18.301622734 +0000 UTC m=+2658.565087681"
Oct 30 19:17:19.300112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:19.297252    5065 generic.go:356] "Generic (PLEG): container finished" podID="ebd97c7a-da68-4d65-8804-a0c13ab96e37" containerID="adec58811b6f325c32039c84b53b008cb7e0afc23e49044bcf36f92c4a861619" exitCode=0
Oct 30 19:17:19.300112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:19.297365    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6" event={"ID":"ebd97c7a-da68-4d65-8804-a0c13ab96e37","Type":"ContainerDied","Data":"adec58811b6f325c32039c84b53b008cb7e0afc23e49044bcf36f92c4a861619"}
Oct 30 19:17:20.521941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.521917    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:20.631290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.631199    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\") pod \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") "
Oct 30 19:17:20.631497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.631480    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\") pod \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") "
Oct 30 19:17:20.631930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.631591    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\" (UniqueName: \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\") pod \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\" (UID: \"ebd97c7a-da68-4d65-8804-a0c13ab96e37\") "
Oct 30 19:17:20.631930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.631826    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca" pod "ebd97c7a-da68-4d65-8804-a0c13ab96e37" (UID: "ebd97c7a-da68-4d65-8804-a0c13ab96e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:20.632603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.632579    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:20.633786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.633766    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca" pod "ebd97c7a-da68-4d65-8804-a0c13ab96e37" (UID: "ebd97c7a-da68-4d65-8804-a0c13ab96e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:20.633871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.633849    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74" pod "ebd97c7a-da68-4d65-8804-a0c13ab96e37" (UID: "ebd97c7a-da68-4d65-8804-a0c13ab96e37"). InnerVolumeSpecName "kube-api-access-89w74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:20.734387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.734361    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w74\" (UniqueName: \"kubernetes.io/projected/ebd97c7a-da68-4d65-8804-a0c13ab96e37-kube-api-access-89w74\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:20.734387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:20.734384    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd97c7a-da68-4d65-8804-a0c13ab96e37-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.330966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:21.330931    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"
Oct 30 19:17:21.331198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:21.330958    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6" event={"ID":"ebd97c7a-da68-4d65-8804-a0c13ab96e37","Type":"ContainerDied","Data":"21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a"}
Oct 30 19:17:21.331198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:21.331040    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21727a52cd1b90c837f1ca585cdb54f69559bd1af8d62cf8966e0e5aae4e238a"
Oct 30 19:17:23.058856 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:23.058823    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:17:46.599940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:46.599894    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:17:46.600598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:46.600196    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy" containerID="cri-o://a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3" gracePeriod=30
Oct 30 19:17:52.014933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.014902    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:17:52.057211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057170    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057224    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057251    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057283    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057485    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057608    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057702    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057773    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:52.057873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057785    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.057873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057856    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.057982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057894    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.057982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057948    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.058222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.057994    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\") pod \"6dcc55db-906b-464c-98d4-18c6a28f32a0\" (UID: \"6dcc55db-906b-464c-98d4-18c6a28f32a0\") "
Oct 30 19:17:52.058222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058066    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.058793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058768    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.058793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058799    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.058962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058814    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.058962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058829    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.058962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.058845    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcc55db-906b-464c-98d4-18c6a28f32a0-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.060155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.060077    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.060867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.060844    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:52.061005 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.060982    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.061296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.061270    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n" pod "6dcc55db-906b-464c-98d4-18c6a28f32a0" (UID: "6dcc55db-906b-464c-98d4-18c6a28f32a0"). InnerVolumeSpecName "kube-api-access-hrj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.062400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.062378    5065 generic.go:356] "Generic (PLEG): container finished" podID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerID="a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3" exitCode=0
Oct 30 19:17:52.062518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.062438    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"
Oct 30 19:17:52.062518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.062465    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" event={"ID":"6dcc55db-906b-464c-98d4-18c6a28f32a0","Type":"ContainerDied","Data":"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"}
Oct 30 19:17:52.062641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.062522    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2" event={"ID":"6dcc55db-906b-464c-98d4-18c6a28f32a0","Type":"ContainerDied","Data":"87a8e1ccc0cd648369ff4cb9ba1b4106d360b26894bc2354e3630fa4b3e090e3"}
Oct 30 19:17:52.062641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.062548    5065 scope.go:122] "RemoveContainer" containerID="a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"
Oct 30 19:17:52.114633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.114614    5065 scope.go:122] "RemoveContainer" containerID="a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"
Oct 30 19:17:52.115057 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:52.115026    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3\": container with ID starting with a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3 not found: ID does not exist" containerID="a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"
Oct 30 19:17:52.115211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.115068    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3"} err="failed to get container status \"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3\": rpc error: code = NotFound desc = could not find container \"a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3\": container with ID starting with a4da11f3622cea927d7e6601e5d6294f2f7a14511d91e95c08f58e2f3587aed3 not found: ID does not exist"
Oct 30 19:17:52.136140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.136068    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:17:52.150155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.150045    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8lcb2"]
Oct 30 19:17:52.155318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.155296    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:17:52.160763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.160736    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrj9n\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-kube-api-access-hrj9n\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.160849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.160768    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.160849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.160786    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.160849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.160800    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcc55db-906b-464c-98d4-18c6a28f32a0-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.195054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:52.195022    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:17:53.074795 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:53.074758    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:17:53.894510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:53.894479    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" path="/var/lib/kubelet/pods/6dcc55db-906b-464c-98d4-18c6a28f32a0/volumes"
Oct 30 19:17:55.033477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:55.033444    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"]
Oct 30 19:17:55.038626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:55.038598    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58a8e268-t8bb8"]
Oct 30 19:17:55.898582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:55.898534    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfa8120-bf52-46b3-b1aa-f120a0513842" path="/var/lib/kubelet/pods/9bfa8120-bf52-46b3-b1aa-f120a0513842/volumes"
Oct 30 19:17:57.068211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:57.068161    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"]
Oct 30 19:17:57.149592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:57.149548    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aee02a-fztm5"]
Oct 30 19:17:57.190345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:57.190301    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" podUID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" containerName="payload-processing" containerID="cri-o://1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2" gracePeriod=25
Oct 30 19:17:57.223293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:57.223164    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerName="payload-processing" containerID="cri-o://f296efdc520b4b92dbef0a1eecb2d99c9b73227b7c166a7766c34d98d1770630" gracePeriod=25
Oct 30 19:17:57.858354 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:17:57.841665    5065 logging.go:55] [core] [Channel #9813 SubChannel #9814]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.97:50052", ServerName: "10.132.1.97:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.97:50052: connect: connection refused"
Oct 30 19:17:58.109820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.097655    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f25bcbf-61ed-48d7-a70c-a3fdd2938c10" path="/var/lib/kubelet/pods/7f25bcbf-61ed-48d7-a70c-a3fdd2938c10/volumes"
Oct 30 19:17:58.187519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.187424    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:17:58.308337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.307135    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\") pod \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") "
Oct 30 19:17:58.308337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.307190    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\") pod \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") "
Oct 30 19:17:58.308337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.307286    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\") pod \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") "
Oct 30 19:17:58.308337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.307341    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\") pod \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\" (UID: \"6d8ab131-db2a-441c-a81f-9e283ef92bbe\") "
Oct 30 19:17:58.308733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.308487    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume" pod "6d8ab131-db2a-441c-a81f-9e283ef92bbe" (UID: "6d8ab131-db2a-441c-a81f-9e283ef92bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:58.310143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.308880    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume" pod "6d8ab131-db2a-441c-a81f-9e283ef92bbe" (UID: "6d8ab131-db2a-441c-a81f-9e283ef92bbe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:58.318018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.317988    5065 generic.go:356] "Generic (PLEG): container finished" podID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" containerID="1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2" exitCode=0
Oct 30 19:17:58.318278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.318257    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" event={"ID":"6d8ab131-db2a-441c-a81f-9e283ef92bbe","Type":"ContainerDied","Data":"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"}
Oct 30 19:17:58.318376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.318314    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx" event={"ID":"6d8ab131-db2a-441c-a81f-9e283ef92bbe","Type":"ContainerDied","Data":"593ca28f7fe2b5ae3863118ad1772f09a15a5b639eb2ae8f98b6e9d95f6688a6"}
Oct 30 19:17:58.318376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.318349    5065 scope.go:122] "RemoveContainer" containerID="1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"
Oct 30 19:17:58.318376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.318364    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"
Oct 30 19:17:58.325669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.325601    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" event={"ID":"a0af92be-b39b-4a2a-b760-73e4986ce05a","Type":"ContainerDied","Data":"f296efdc520b4b92dbef0a1eecb2d99c9b73227b7c166a7766c34d98d1770630"}
Oct 30 19:17:58.325829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.325565    5065 generic.go:356] "Generic (PLEG): container finished" podID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerID="f296efdc520b4b92dbef0a1eecb2d99c9b73227b7c166a7766c34d98d1770630" exitCode=0
Oct 30 19:17:58.330961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.330914    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds" pod "6d8ab131-db2a-441c-a81f-9e283ef92bbe" (UID: "6d8ab131-db2a-441c-a81f-9e283ef92bbe"). InnerVolumeSpecName "kube-api-access-vw9ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:58.357566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.355664    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials" pod "6d8ab131-db2a-441c-a81f-9e283ef92bbe" (UID: "6d8ab131-db2a-441c-a81f-9e283ef92bbe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:58.387945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.387926    5065 scope.go:122] "RemoveContainer" containerID="1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"
Oct 30 19:17:58.388048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.387960    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:17:58.388709 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:17:58.388683    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2\": container with ID starting with 1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2 not found: ID does not exist" containerID="1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"
Oct 30 19:17:58.388765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.388723    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2"} err="failed to get container status \"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2\": rpc error: code = NotFound desc = could not find container \"1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2\": container with ID starting with 1b7b207d8bbf5165d3c355a7f23f2c77362754e05e077bc154c00279f0dcdfc2 not found: ID does not exist"
Oct 30 19:17:58.412691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.409923    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.412691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.409956    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8ab131-db2a-441c-a81f-9e283ef92bbe-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.412691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.409993    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.412691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.410007    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9ds\" (UniqueName: \"kubernetes.io/projected/6d8ab131-db2a-441c-a81f-9e283ef92bbe-kube-api-access-vw9ds\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.510756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.510720    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\") pod \"a0af92be-b39b-4a2a-b760-73e4986ce05a\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") "
Oct 30 19:17:58.513937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.511405    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume" pod "a0af92be-b39b-4a2a-b760-73e4986ce05a" (UID: "a0af92be-b39b-4a2a-b760-73e4986ce05a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:58.515605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.515580    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\") pod \"a0af92be-b39b-4a2a-b760-73e4986ce05a\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") "
Oct 30 19:17:58.515868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.515847    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\") pod \"a0af92be-b39b-4a2a-b760-73e4986ce05a\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") "
Oct 30 19:17:58.516167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.516146    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\") pod \"a0af92be-b39b-4a2a-b760-73e4986ce05a\" (UID: \"a0af92be-b39b-4a2a-b760-73e4986ce05a\") "
Oct 30 19:17:58.517192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.517168    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.517484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.517463    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume" pod "a0af92be-b39b-4a2a-b760-73e4986ce05a" (UID: "a0af92be-b39b-4a2a-b760-73e4986ce05a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:58.520467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.520446    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials" pod "a0af92be-b39b-4a2a-b760-73e4986ce05a" (UID: "a0af92be-b39b-4a2a-b760-73e4986ce05a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:58.520895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.520870    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p" pod "a0af92be-b39b-4a2a-b760-73e4986ce05a" (UID: "a0af92be-b39b-4a2a-b760-73e4986ce05a"). InnerVolumeSpecName "kube-api-access-lnn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:58.618928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.618892    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0af92be-b39b-4a2a-b760-73e4986ce05a-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.619206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.619068    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.619206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.619198    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnn9p\" (UniqueName: \"kubernetes.io/projected/a0af92be-b39b-4a2a-b760-73e4986ce05a-kube-api-access-lnn9p\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:58.651814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.651783    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:17:58.660771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.660742    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbpf9hx"]
Oct 30 19:17:58.841636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:58.841593    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.97:50052\" within 1s: context deadline exceeded"
Oct 30 19:17:59.338166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.338132    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf" event={"ID":"a0af92be-b39b-4a2a-b760-73e4986ce05a","Type":"ContainerDied","Data":"b4f7c797fa1819ff953fcd0f065459f8ad0eb0c2405cfa79b5d1a45345a70d64"}
Oct 30 19:17:59.338685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.338191    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"
Oct 30 19:17:59.338685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.338194    5065 scope.go:122] "RemoveContainer" containerID="f296efdc520b4b92dbef0a1eecb2d99c9b73227b7c166a7766c34d98d1770630"
Oct 30 19:17:59.372030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.371999    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:17:59.377879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.377856    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59m5jpf"]
Oct 30 19:17:59.937204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.931296    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" path="/var/lib/kubelet/pods/6d8ab131-db2a-441c-a81f-9e283ef92bbe/volumes"
Oct 30 19:17:59.937204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:17:59.932164    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" path="/var/lib/kubelet/pods/a0af92be-b39b-4a2a-b760-73e4986ce05a/volumes"
Oct 30 19:18:03.901444 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:03.901417    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:18:03.910458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:03.910429    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:18:03.987754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:03.987722    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:18:03.996780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:03.996754    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:18:06.096568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:06.096535    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"]
Oct 30 19:18:06.108289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:06.108259    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d134ea-qcf2z"]
Oct 30 19:18:07.895354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:07.895319    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7633aea0-4d0f-494f-ade4-ffdcc05528db" path="/var/lib/kubelet/pods/7633aea0-4d0f-494f-ade4-ffdcc05528db/volumes"
Oct 30 19:18:09.044317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:09.044282    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"]
Oct 30 19:18:09.049380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:09.049350    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4c1e80-7sw9h"]
Oct 30 19:18:09.896188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:09.896148    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea" path="/var/lib/kubelet/pods/fa4e9deb-46e6-48d0-b32c-dd75cff6a8ea/volumes"
Oct 30 19:18:15.292443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:15.292405    5065 scope.go:122] "RemoveContainer" containerID="0f7b9bfc37df33c69cb7236ea8df4a9d8197f597acddbf1024f37fdda34cbff7"
Oct 30 19:18:15.304210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:15.304186    5065 scope.go:122] "RemoveContainer" containerID="b10ea5bcd2ccaca4a9d1ddba212dbb84fcd7daf99a1f27754ccdc6f9a8e10599"
Oct 30 19:18:15.315239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:15.315219    5065 scope.go:122] "RemoveContainer" containerID="6ac96c7ff3d6e75392bd1796671581c926d62db6f217e7113239601b933eae04"
Oct 30 19:18:15.326023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:15.325996    5065 scope.go:122] "RemoveContainer" containerID="97b033979a4dfaae6087aa3e196ad5c94a5cc0c36309d987546698e1202c50cd"
Oct 30 19:18:17.029761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:17.029730    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"]
Oct 30 19:18:17.034569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:17.034546    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65207ead-xvhmn"]
Oct 30 19:18:17.896244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:17.896211    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d" path="/var/lib/kubelet/pods/1bc2a93e-9613-4fe8-8aee-d6a2ab6d086d/volumes"
Oct 30 19:18:23.058178 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:18:23.058134    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:18:34.818589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.818549    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"]
Oct 30 19:18:34.819415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819387    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" containerName="payload-processing"
Oct 30 19:18:34.819415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819414    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" containerName="payload-processing"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819428    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ebd97c7a-da68-4d65-8804-a0c13ab96e37" containerName="revoke-keys"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819433    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd97c7a-da68-4d65-8804-a0c13ab96e37" containerName="revoke-keys"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819452    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819457    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819470    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerName="payload-processing"
Oct 30 19:18:34.819559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819475    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerName="payload-processing"
Oct 30 19:18:34.819899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819588    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0af92be-b39b-4a2a-b760-73e4986ce05a" containerName="payload-processing"
Oct 30 19:18:34.819899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819606    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8ab131-db2a-441c-a81f-9e283ef92bbe" containerName="payload-processing"
Oct 30 19:18:34.819899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819611    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd97c7a-da68-4d65-8804-a0c13ab96e37" containerName="revoke-keys"
Oct 30 19:18:34.819899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.819618    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcc55db-906b-464c-98d4-18c6a28f32a0" containerName="istio-proxy"
Oct 30 19:18:34.823607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.823589    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:34.834630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.834605    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:18:34.834778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.834681    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:34.841149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:34.841124    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"]
Oct 30 19:18:35.001356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.001314    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.001628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.001608    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptghs\" (UniqueName: \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.001909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.001837    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.103969    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.104047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.104008    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptghs\" (UniqueName: \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.104318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.104122    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.104684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.104663    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.106835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.106804    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.118130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.118101    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptghs\" (UniqueName: \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\") pod \"maas-api-revoke-sub-6035f83c-w7ks6\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.140072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.140048    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:35.440132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.440105    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"]
Oct 30 19:18:35.441920 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:18:35.441877    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda687236_bdfb_4726_9ca5_cedc3962790e.slice/crio-cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b WatchSource:0}: Error finding container cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b: Status 404 returned error can't find the container with id cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b
Oct 30 19:18:35.895619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.895582    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6" event={"ID":"da687236-bdfb-4726-9ca5-cedc3962790e","Type":"ContainerStarted","Data":"1eff6989007d7bda687fc1fdafb7c707dd0ff412ee95c1995bc6650eb44c89bb"}
Oct 30 19:18:35.896059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.895649    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6" event={"ID":"da687236-bdfb-4726-9ca5-cedc3962790e","Type":"ContainerStarted","Data":"cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b"}
Oct 30 19:18:35.935439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:35.935373    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6" podStartSLOduration=1.935355248 podStartE2EDuration="1.935355248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:35.930701942 +0000 UTC m=+2736.194166892" watchObservedRunningTime="2026-10-30 19:18:35.935355248 +0000 UTC m=+2736.198820199"
Oct 30 19:18:36.314567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.314531    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"]
Oct 30 19:18:36.320420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.320396    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.333797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.333772    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"]
Oct 30 19:18:36.436154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.436126    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.436296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.436163    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.436467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.436445    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflsl\" (UniqueName: \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.537738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.537701    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.537738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.537748    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.538021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.538004    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflsl\" (UniqueName: \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.538553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.538473    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.540296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.540275    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.553158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.553132    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflsl\" (UniqueName: \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\") pod \"maas-api-revoke-sub-59ccd0c0-rdjgm\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.638733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.638670    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:36.906194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.905771    5065 generic.go:356] "Generic (PLEG): container finished" podID="da687236-bdfb-4726-9ca5-cedc3962790e" containerID="1eff6989007d7bda687fc1fdafb7c707dd0ff412ee95c1995bc6650eb44c89bb" exitCode=0
Oct 30 19:18:36.906194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.905854    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6" event={"ID":"da687236-bdfb-4726-9ca5-cedc3962790e","Type":"ContainerDied","Data":"1eff6989007d7bda687fc1fdafb7c707dd0ff412ee95c1995bc6650eb44c89bb"}
Oct 30 19:18:36.931656 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:18:36.931629    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1712b45c_892e_4608_b122_ea8202e3f191.slice/crio-2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c WatchSource:0}: Error finding container 2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c: Status 404 returned error can't find the container with id 2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c
Oct 30 19:18:36.934943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:36.934899    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"]
Oct 30 19:18:37.920972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:37.920938    5065 generic.go:356] "Generic (PLEG): container finished" podID="1712b45c-892e-4608-b122-ea8202e3f191" containerID="6b827a5094445dba5c4621d00017b4231878732e4b87dfcc45a87f190b87ffb0" exitCode=0
Oct 30 19:18:37.921411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:37.921014    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm" event={"ID":"1712b45c-892e-4608-b122-ea8202e3f191","Type":"ContainerDied","Data":"6b827a5094445dba5c4621d00017b4231878732e4b87dfcc45a87f190b87ffb0"}
Oct 30 19:18:37.921411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:37.921077    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm" event={"ID":"1712b45c-892e-4608-b122-ea8202e3f191","Type":"ContainerStarted","Data":"2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c"}
Oct 30 19:18:38.178157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.174208    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:38.273070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.271670    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\") pod \"da687236-bdfb-4726-9ca5-cedc3962790e\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") "
Oct 30 19:18:38.273070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.272170    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\" (UniqueName: \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\") pod \"da687236-bdfb-4726-9ca5-cedc3962790e\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") "
Oct 30 19:18:38.273070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.272296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\") pod \"da687236-bdfb-4726-9ca5-cedc3962790e\" (UID: \"da687236-bdfb-4726-9ca5-cedc3962790e\") "
Oct 30 19:18:38.277123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.273505    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca" pod "da687236-bdfb-4726-9ca5-cedc3962790e" (UID: "da687236-bdfb-4726-9ca5-cedc3962790e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:38.282871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.278491    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca" pod "da687236-bdfb-4726-9ca5-cedc3962790e" (UID: "da687236-bdfb-4726-9ca5-cedc3962790e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:38.290125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.283570    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs" pod "da687236-bdfb-4726-9ca5-cedc3962790e" (UID: "da687236-bdfb-4726-9ca5-cedc3962790e"). InnerVolumeSpecName "kube-api-access-ptghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:38.374592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.374559    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptghs\" (UniqueName: \"kubernetes.io/projected/da687236-bdfb-4726-9ca5-cedc3962790e-kube-api-access-ptghs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:38.374592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.374592    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:38.374758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.374608    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da687236-bdfb-4726-9ca5-cedc3962790e-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:38.933297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.933264    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"
Oct 30 19:18:38.933297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.933291    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6" event={"ID":"da687236-bdfb-4726-9ca5-cedc3962790e","Type":"ContainerDied","Data":"cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b"}
Oct 30 19:18:38.933842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:38.933333    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae536ebc17c86d3912264269b3a84876911d9c360a6f554608311dfd3b99c0b"
Oct 30 19:18:39.171002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.163146    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:39.310872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.310843    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\" (UniqueName: \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\") pod \"1712b45c-892e-4608-b122-ea8202e3f191\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") "
Oct 30 19:18:39.311139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.310928    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\") pod \"1712b45c-892e-4608-b122-ea8202e3f191\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") "
Oct 30 19:18:39.311238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.311154    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\") pod \"1712b45c-892e-4608-b122-ea8202e3f191\" (UID: \"1712b45c-892e-4608-b122-ea8202e3f191\") "
Oct 30 19:18:39.311381 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.311355    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca" pod "1712b45c-892e-4608-b122-ea8202e3f191" (UID: "1712b45c-892e-4608-b122-ea8202e3f191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:39.312135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.312108    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:39.313340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.313315    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl" pod "1712b45c-892e-4608-b122-ea8202e3f191" (UID: "1712b45c-892e-4608-b122-ea8202e3f191"). InnerVolumeSpecName "kube-api-access-xflsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:39.313473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.313454    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca" pod "1712b45c-892e-4608-b122-ea8202e3f191" (UID: "1712b45c-892e-4608-b122-ea8202e3f191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:39.413462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.413429    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflsl\" (UniqueName: \"kubernetes.io/projected/1712b45c-892e-4608-b122-ea8202e3f191-kube-api-access-xflsl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:39.413462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.413461    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712b45c-892e-4608-b122-ea8202e3f191-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:39.943386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.943360    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"
Oct 30 19:18:39.943781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.943397    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm" event={"ID":"1712b45c-892e-4608-b122-ea8202e3f191","Type":"ContainerDied","Data":"2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c"}
Oct 30 19:18:39.943781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:39.943450    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab5ca16d476c480352888832913fe8c7bbfb851c6c859eafd4bc6a4a01a9b8c"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.460916    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"]
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462477    5065 cpu_manager.go:414] "containerMap: removing container" podUID="da687236-bdfb-4726-9ca5-cedc3962790e" containerName="revoke-keys"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462499    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da687236-bdfb-4726-9ca5-cedc3962790e" containerName="revoke-keys"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462525    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1712b45c-892e-4608-b122-ea8202e3f191" containerName="revoke-keys"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462533    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1712b45c-892e-4608-b122-ea8202e3f191" containerName="revoke-keys"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462769    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="da687236-bdfb-4726-9ca5-cedc3962790e" containerName="revoke-keys"
Oct 30 19:18:46.465117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.462785    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1712b45c-892e-4608-b122-ea8202e3f191" containerName="revoke-keys"
Oct 30 19:18:46.477718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.470677    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.483003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.482973    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"]
Oct 30 19:18:46.508357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.508326    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:46.508509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.508446    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:18:46.563526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.563487    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.563712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.563545    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.563823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.563803    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrns\" (UniqueName: \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.668442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.668396    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.671531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.671505    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.676871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.676842    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrns\" (UniqueName: \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.725197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.671426    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.725197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.692713    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.774112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.763258    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrns\" (UniqueName: \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\") pod \"maas-api-revoke-sub-a3028cb3-f8zk6\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:46.810500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:46.810462    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:47.437118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:47.433945    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"]
Oct 30 19:18:47.505967 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:18:47.499569    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982b884d_1b16_4757_a607_6545820d6d10.slice/crio-bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27 WatchSource:0}: Error finding container bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27: Status 404 returned error can't find the container with id bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27
Oct 30 19:18:48.148480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:48.148441    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6" event={"ID":"982b884d-1b16-4757-a607-6545820d6d10","Type":"ContainerStarted","Data":"bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27"}
Oct 30 19:18:49.158714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.158679    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6" event={"ID":"982b884d-1b16-4757-a607-6545820d6d10","Type":"ContainerStarted","Data":"61f72ab380b8665d37b539d09117e4047062e10520350fb6e4627bfb397da808"}
Oct 30 19:18:49.210930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.210885    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6" podStartSLOduration=3.210870619 podStartE2EDuration="3.210870619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:49.205914778 +0000 UTC m=+2749.469379727" watchObservedRunningTime="2026-10-30 19:18:49.210870619 +0000 UTC m=+2749.474335611"
Oct 30 19:18:49.219637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.219613    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"]
Oct 30 19:18:49.225072 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.225040    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.239363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.239330    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"]
Oct 30 19:18:49.342586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.342554    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.342749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.342729    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.342863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.342844    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbklr\" (UniqueName: \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.444862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.444801    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.444993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.444879    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbklr\" (UniqueName: \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.445035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.445024    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.445487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.445456    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.447548 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.447521    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.458246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.458225    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbklr\" (UniqueName: \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\") pod \"maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.574046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.574019    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:49.916272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:49.916248    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"]
Oct 30 19:18:50.169165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:50.169056    5065 generic.go:356] "Generic (PLEG): container finished" podID="982b884d-1b16-4757-a607-6545820d6d10" containerID="61f72ab380b8665d37b539d09117e4047062e10520350fb6e4627bfb397da808" exitCode=0
Oct 30 19:18:50.169582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:50.169200    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6" event={"ID":"982b884d-1b16-4757-a607-6545820d6d10","Type":"ContainerDied","Data":"61f72ab380b8665d37b539d09117e4047062e10520350fb6e4627bfb397da808"}
Oct 30 19:18:50.170905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:50.170883    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r" event={"ID":"c13d6882-58e8-4f10-bf47-eb9f3ce9726b","Type":"ContainerStarted","Data":"b3c1b5d3d00a756c84dca2f1550071c21242e09c5aca8ef99973630a7f114f26"}
Oct 30 19:18:51.185505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.185471    5065 generic.go:356] "Generic (PLEG): container finished" podID="c13d6882-58e8-4f10-bf47-eb9f3ce9726b" containerID="8004e381ce076ea4d139f91439902ee46dbe050be644412630457f7b935caa10" exitCode=0
Oct 30 19:18:51.186115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.186072    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r" event={"ID":"c13d6882-58e8-4f10-bf47-eb9f3ce9726b","Type":"ContainerDied","Data":"8004e381ce076ea4d139f91439902ee46dbe050be644412630457f7b935caa10"}
Oct 30 19:18:51.399484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.399458    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:51.663167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.659678    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\" (UniqueName: \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\") pod \"982b884d-1b16-4757-a607-6545820d6d10\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") "
Oct 30 19:18:51.663167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.659750    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\") pod \"982b884d-1b16-4757-a607-6545820d6d10\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") "
Oct 30 19:18:51.663167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.659801    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\") pod \"982b884d-1b16-4757-a607-6545820d6d10\" (UID: \"982b884d-1b16-4757-a607-6545820d6d10\") "
Oct 30 19:18:51.725973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.725918    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca" pod "982b884d-1b16-4757-a607-6545820d6d10" (UID: "982b884d-1b16-4757-a607-6545820d6d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.763018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.762979    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982b884d-1b16-4757-a607-6545820d6d10-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.792817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.792760    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca" pod "982b884d-1b16-4757-a607-6545820d6d10" (UID: "982b884d-1b16-4757-a607-6545820d6d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:51.809576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.809535    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns" pod "982b884d-1b16-4757-a607-6545820d6d10" (UID: "982b884d-1b16-4757-a607-6545820d6d10"). InnerVolumeSpecName "kube-api-access-6nrns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.865127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.865040    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrns\" (UniqueName: \"kubernetes.io/projected/982b884d-1b16-4757-a607-6545820d6d10-kube-api-access-6nrns\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.865127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:51.865100    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982b884d-1b16-4757-a607-6545820d6d10-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:52.197515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.196827    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"
Oct 30 19:18:52.197515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.196853    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6" event={"ID":"982b884d-1b16-4757-a607-6545820d6d10","Type":"ContainerDied","Data":"bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27"}
Oct 30 19:18:52.197515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.196906    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad11eea992365d1e387eb1952f1ac9c489d9f1f7e091fb130396342f145ca27"
Oct 30 19:18:52.390067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.390043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:52.511655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.511619    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\") pod \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") "
Oct 30 19:18:52.511894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.511875    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\") pod \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") "
Oct 30 19:18:52.511996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.511975    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca" pod "c13d6882-58e8-4f10-bf47-eb9f3ce9726b" (UID: "c13d6882-58e8-4f10-bf47-eb9f3ce9726b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:52.512063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.512050    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\" (UniqueName: \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\") pod \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\" (UID: \"c13d6882-58e8-4f10-bf47-eb9f3ce9726b\") "
Oct 30 19:18:52.512716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.512638    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:52.514271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.514243    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca" pod "c13d6882-58e8-4f10-bf47-eb9f3ce9726b" (UID: "c13d6882-58e8-4f10-bf47-eb9f3ce9726b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:52.514448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.514426    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr" pod "c13d6882-58e8-4f10-bf47-eb9f3ce9726b" (UID: "c13d6882-58e8-4f10-bf47-eb9f3ce9726b"). InnerVolumeSpecName "kube-api-access-mbklr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:52.614586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.614517    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:52.614586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:52.614543    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbklr\" (UniqueName: \"kubernetes.io/projected/c13d6882-58e8-4f10-bf47-eb9f3ce9726b-kube-api-access-mbklr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:53.101249 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:18:53.101207    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:18:53.210271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:53.210230    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"
Oct 30 19:18:53.210760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:53.210269    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r" event={"ID":"c13d6882-58e8-4f10-bf47-eb9f3ce9726b","Type":"ContainerDied","Data":"b3c1b5d3d00a756c84dca2f1550071c21242e09c5aca8ef99973630a7f114f26"}
Oct 30 19:18:53.210893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:53.210787    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3c1b5d3d00a756c84dca2f1550071c21242e09c5aca8ef99973630a7f114f26"
Oct 30 19:18:55.032592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.032517    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"]
Oct 30 19:18:55.038104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.038061    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1dba16-xms97"]
Oct 30 19:18:55.042886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.042862    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"]
Oct 30 19:18:55.046904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.046881    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53bed78f-lf6k9"]
Oct 30 19:18:55.894961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.894916    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c056dd9-a767-4342-b567-18b7ac210ef9" path="/var/lib/kubelet/pods/5c056dd9-a767-4342-b567-18b7ac210ef9/volumes"
Oct 30 19:18:55.898915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:55.898885    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4cd31c-d1fe-4921-a1e6-9daad77f0731" path="/var/lib/kubelet/pods/7b4cd31c-d1fe-4921-a1e6-9daad77f0731/volumes"
Oct 30 19:18:57.630108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.630045    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631388    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c13d6882-58e8-4f10-bf47-eb9f3ce9726b" containerName="revoke-keys"
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631415    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13d6882-58e8-4f10-bf47-eb9f3ce9726b" containerName="revoke-keys"
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631456    5065 cpu_manager.go:414] "containerMap: removing container" podUID="982b884d-1b16-4757-a607-6545820d6d10" containerName="revoke-keys"
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631464    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982b884d-1b16-4757-a607-6545820d6d10" containerName="revoke-keys"
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631700    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13d6882-58e8-4f10-bf47-eb9f3ce9726b" containerName="revoke-keys"
Oct 30 19:18:57.635141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.631722    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="982b884d-1b16-4757-a607-6545820d6d10" containerName="revoke-keys"
Oct 30 19:18:57.641404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.641378    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.646816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.646786    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:18:57.649149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649079    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-56125a628169\""
Oct 30 19:18:57.649291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649254    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-e2e--a67c8769-dockercfg-czgjj\""
Oct 30 19:18:57.649371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649294    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-675c1bdb5fa1\""
Oct 30 19:18:57.649371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649329    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-b77e998367e9\""
Oct 30 19:18:57.649563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649430    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"kube-root-ca.crt\""
Oct 30 19:18:57.649563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649490    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"routing-overlay\""
Oct 30 19:18:57.649662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649576    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"openshift-service-ca.crt\""
Oct 30 19:18:57.649903 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.649883    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c6027\""
Oct 30 19:18:57.770823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.770767    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.771025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.770890    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.771155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.771021    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfx4\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.771233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.771180    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.872855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.872824    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.873140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.873120    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.873262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.873177    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.873262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.873251    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfx4\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.873764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.873745    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.873847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.873812    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.875559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.875540    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.887736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.887677    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfx4\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:57.965752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:57.965722    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:58.282175 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:18:58.282138    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9001629e_1c47_45d5_983c_86fb03663a5d.slice/crio-b5d3784f781b6b347e070696b495fc9e183d8df0fc251bfe1fea76e06aaff2ba WatchSource:0}: Error finding container b5d3784f781b6b347e070696b495fc9e183d8df0fc251bfe1fea76e06aaff2ba: Status 404 returned error can't find the container with id b5d3784f781b6b347e070696b495fc9e183d8df0fc251bfe1fea76e06aaff2ba
Oct 30 19:18:58.284470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:58.284443    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:18:58.324046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:58.324015    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" event={"ID":"9001629e-1c47-45d5-983c-86fb03663a5d","Type":"ContainerStarted","Data":"b5d3784f781b6b347e070696b495fc9e183d8df0fc251bfe1fea76e06aaff2ba"}
Oct 30 19:18:58.791220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:58.791178    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:18:59.388953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:59.388914    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" event={"ID":"9001629e-1c47-45d5-983c-86fb03663a5d","Type":"ContainerStarted","Data":"4ae2f43fd36be192b34c4d11be282d24d73644d9405a25e05e00defb80429a8e"}
Oct 30 19:18:59.391564 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:59.391540    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:18:59.443417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:18:59.443355    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" podStartSLOduration=2.443337778 podStartE2EDuration="2.443337778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:59.425473428 +0000 UTC m=+2759.688938377" watchObservedRunningTime="2026-10-30 19:18:59.443337778 +0000 UTC m=+2759.706802726"
Oct 30 19:19:00.936233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.936204    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:19:00.941578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.941551    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:00.951569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.951544    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"kube-root-ca.crt\""
Oct 30 19:19:00.951743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.951647    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-e2e--ebe4ecff-dockercfg-jcrz9\""
Oct 30 19:19:00.951835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.951778    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a717c7\""
Oct 30 19:19:00.951914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.951883    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"agc-cred-8b1ebed38d09\""
Oct 30 19:19:00.951985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.951957    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"openshift-service-ca.crt\""
Oct 30 19:19:00.952117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.952073    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"routing-overlay\""
Oct 30 19:19:00.958400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:00.958031    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:19:01.137385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.137332    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2z9\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.137902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.137869    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.138506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.138468    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.138769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.138751    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.260667    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.260725    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.260850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2z9\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.260992    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.261882    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.266164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.262059    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.470404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.460068    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.808895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.806171    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2z9\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:01.820537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.819194    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:19:01.887117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:01.876799    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:02.025763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.025725    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.091746    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\""
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.092848    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-j6kxc\""
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.092939    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-940889465afe\""
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.093053    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\""
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.093192    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\""
Oct 30 19:19:02.105111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.093285    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\""
Oct 30 19:19:02.283663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.283622    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:19:02.301293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.300342    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.302944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.302915    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmmt\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.303285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.303261    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.304353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.304332    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.406633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.406595    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.406806 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.406716    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmmt\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.407155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.407135    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.407264 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.407184    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.408141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.408044    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.408274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.408169    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.410279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.410252    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.425392 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.423808    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmmt\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.468650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.468625    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:02.843419 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.843378    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" event={"ID":"c9f7e091-31ca-493f-b301-8b35fe6b2620","Type":"ContainerStarted","Data":"6871e51754f7cb0da4457a3dc3ca2b6b1442b55885a43d5fe9f90cb681a929b5"}
Oct 30 19:19:02.880542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:02.880509    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:19:03.204625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.204569    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:19:03.791569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.791531    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerName="payload-processing" containerID="cri-o://c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea" gracePeriod=25
Oct 30 19:19:03.830138 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:03.830067    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f3267_6025_4f61_8207_3bdea2b85bee.slice/crio-c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:19:03.854407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.854381    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" event={"ID":"c9f7e091-31ca-493f-b301-8b35fe6b2620","Type":"ContainerStarted","Data":"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"}
Oct 30 19:19:03.854557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.854482    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:03.856305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.856281    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" event={"ID":"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f","Type":"ContainerStarted","Data":"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"}
Oct 30 19:19:03.856435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.856324    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" event={"ID":"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f","Type":"ContainerStarted","Data":"f5797ef6bfecc33274abb0400b4c8eaf724491456b85d0294b255da4c3833ffa"}
Oct 30 19:19:03.856435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.856349    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:03.880001 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.879956    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" podStartSLOduration=3.879943267 podStartE2EDuration="3.879943267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:03.877219291 +0000 UTC m=+2764.140684258" watchObservedRunningTime="2026-10-30 19:19:03.879943267 +0000 UTC m=+2764.143408214"
Oct 30 19:19:03.926108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:03.922533    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" podStartSLOduration=2.922519286 podStartE2EDuration="2.922519286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:03.915811796 +0000 UTC m=+2764.179276745" watchObservedRunningTime="2026-10-30 19:19:03.922519286 +0000 UTC m=+2764.185984233"
Oct 30 19:19:04.775152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.774677    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:19:04.797007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.794372    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"]
Oct 30 19:19:04.797007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.795641    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerName="payload-processing"
Oct 30 19:19:04.797007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.795658    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerName="payload-processing"
Oct 30 19:19:04.797007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.795896    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerName="payload-processing"
Oct 30 19:19:04.985470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.985436    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"]
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:04.986550    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.003769    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\") pod \"8e5f3267-6025-4f61-8207-3bdea2b85bee\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") "
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.003932    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\") pod \"8e5f3267-6025-4f61-8207-3bdea2b85bee\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") "
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.004079    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") pod \"8e5f3267-6025-4f61-8207-3bdea2b85bee\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") "
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.004313    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\") pod \"8e5f3267-6025-4f61-8207-3bdea2b85bee\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") "
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.005535    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume" pod "8e5f3267-6025-4f61-8207-3bdea2b85bee" (UID: "8e5f3267-6025-4f61-8207-3bdea2b85bee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.006255    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume" pod "8e5f3267-6025-4f61-8207-3bdea2b85bee" (UID: "8e5f3267-6025-4f61-8207-3bdea2b85bee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.043416    5065 generic.go:356] "Generic (PLEG): container finished" podID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerID="c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea" exitCode=0
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.046002    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.046261    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" event={"ID":"8e5f3267-6025-4f61-8207-3bdea2b85bee","Type":"ContainerDied","Data":"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"}
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.046307    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" event={"ID":"8e5f3267-6025-4f61-8207-3bdea2b85bee","Type":"ContainerDied","Data":"dd52625dff05099371306f59eee477f26a664672b3505240e32355419724b517"}
Oct 30 19:19:05.061157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.046343    5065 scope.go:122] "RemoveContainer" containerID="c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"
Oct 30 19:19:05.095547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.076154    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:19:05.095547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.076673    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:05.138476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.129916    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.138476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.129980    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.138476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.130232    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4kqv\" (UniqueName: \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.185110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.182870    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:05.185110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.183891    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5f3267-6025-4f61-8207-3bdea2b85bee-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:05.330792 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.330690    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials" pod "8e5f3267-6025-4f61-8207-3bdea2b85bee" (UID: "8e5f3267-6025-4f61-8207-3bdea2b85bee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:05.395125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.358812    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:19:05.575836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.537571    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") pod \"8e5f3267-6025-4f61-8207-3bdea2b85bee\" (UID: \"8e5f3267-6025-4f61-8207-3bdea2b85bee\") "
Oct 30 19:19:05.590063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.589984    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:19:05.611112 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:05.579375    5065 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8e5f3267-6025-4f61-8207-3bdea2b85bee/volumes/kubernetes.io~projected/provider-credentials
Oct 30 19:19:05.611112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.608406    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials" pod "8e5f3267-6025-4f61-8207-3bdea2b85bee" (UID: "8e5f3267-6025-4f61-8207-3bdea2b85bee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:05.611112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.583415    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d" pod "8e5f3267-6025-4f61-8207-3bdea2b85bee" (UID: "8e5f3267-6025-4f61-8207-3bdea2b85bee"). InnerVolumeSpecName "kube-api-access-p9f5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:05.611429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.611261    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.88:50052\" within 1s: context deadline exceeded"
Oct 30 19:19:05.611908 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:05.611843    5065 logging.go:55] [core] [Channel #10298 SubChannel #10299]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.88:50052", ServerName: "10.132.1.88:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.88:50052: operation was canceled"
Oct 30 19:19:05.658572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.658530    5065 scope.go:122] "RemoveContainer" containerID="c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"
Oct 30 19:19:05.660183 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.660113    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kqv\" (UniqueName: \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.665418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.665393    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.665642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.665623    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.666415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.666394    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:05.666558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.666543    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9f5d\" (UniqueName: \"kubernetes.io/projected/8e5f3267-6025-4f61-8207-3bdea2b85bee-kube-api-access-p9f5d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:05.667499 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:05.661956    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea\": container with ID starting with c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea not found: ID does not exist" containerID="c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"
Oct 30 19:19:05.667605 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.667512    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea"} err="failed to get container status \"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea\": rpc error: code = NotFound desc = could not find container \"c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea\": container with ID starting with c31b31419b0e6f01fb34839b1be4230e438251119defbc68f6d408d3be05ecea not found: ID does not exist"
Oct 30 19:19:05.668225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.668202    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"]
Oct 30 19:19:05.668430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.668409    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.685598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.673519    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.725038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.686421    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4kqv\" (UniqueName: \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\") pod \"maas-api-revoke-sub-05931ac2-kcdsp\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.725038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.690309    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c27ed4-4bjhg"]
Oct 30 19:19:05.901155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.896734    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dff3ce-aaca-4bad-a0d6-bfad13e2cfff" path="/var/lib/kubelet/pods/97dff3ce-aaca-4bad-a0d6-bfad13e2cfff/volumes"
Oct 30 19:19:05.945412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.945387    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:05.982206 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.982149    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:19:05.987436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:05.987404    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85ffd45dc8-tpp8p"]
Oct 30 19:19:06.373254 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:06.373212    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28d7286_1061_4515_8568_e952ed62ca6c.slice/crio-0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e WatchSource:0}: Error finding container 0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e: Status 404 returned error can't find the container with id 0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e
Oct 30 19:19:06.374065 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:06.374035    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"]
Oct 30 19:19:07.096397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:07.096309    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp" event={"ID":"d28d7286-1061-4515-8568-e952ed62ca6c","Type":"ContainerStarted","Data":"e7e151d663d52a1a572e4d41c49462c82754a52b223403db9747b8ffa2ccf22d"}
Oct 30 19:19:07.096397 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:07.096362    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp" event={"ID":"d28d7286-1061-4515-8568-e952ed62ca6c","Type":"ContainerStarted","Data":"0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e"}
Oct 30 19:19:07.193949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:07.193895    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp" podStartSLOduration=3.193880295 podStartE2EDuration="3.193880295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:07.191370474 +0000 UTC m=+2767.454835436" watchObservedRunningTime="2026-10-30 19:19:07.193880295 +0000 UTC m=+2767.457345242"
Oct 30 19:19:07.901016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:07.900980    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5f3267-6025-4f61-8207-3bdea2b85bee" path="/var/lib/kubelet/pods/8e5f3267-6025-4f61-8207-3bdea2b85bee/volumes"
Oct 30 19:19:08.126513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:08.126474    5065 generic.go:356] "Generic (PLEG): container finished" podID="d28d7286-1061-4515-8568-e952ed62ca6c" containerID="e7e151d663d52a1a572e4d41c49462c82754a52b223403db9747b8ffa2ccf22d" exitCode=0
Oct 30 19:19:08.126914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:08.126533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp" event={"ID":"d28d7286-1061-4515-8568-e952ed62ca6c","Type":"ContainerDied","Data":"e7e151d663d52a1a572e4d41c49462c82754a52b223403db9747b8ffa2ccf22d"}
Oct 30 19:19:09.675778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.675749    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:09.809208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.809140    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\" (UniqueName: \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\") pod \"d28d7286-1061-4515-8568-e952ed62ca6c\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") "
Oct 30 19:19:09.809347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.809304    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\") pod \"d28d7286-1061-4515-8568-e952ed62ca6c\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") "
Oct 30 19:19:09.809466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.809447    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\") pod \"d28d7286-1061-4515-8568-e952ed62ca6c\" (UID: \"d28d7286-1061-4515-8568-e952ed62ca6c\") "
Oct 30 19:19:09.809689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.809660    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca" pod "d28d7286-1061-4515-8568-e952ed62ca6c" (UID: "d28d7286-1061-4515-8568-e952ed62ca6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:09.810105 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.810068    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.811612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.811589    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv" pod "d28d7286-1061-4515-8568-e952ed62ca6c" (UID: "d28d7286-1061-4515-8568-e952ed62ca6c"). InnerVolumeSpecName "kube-api-access-b4kqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:09.811697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.811634    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca" pod "d28d7286-1061-4515-8568-e952ed62ca6c" (UID: "d28d7286-1061-4515-8568-e952ed62ca6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:09.915218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.915177    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4kqv\" (UniqueName: \"kubernetes.io/projected/d28d7286-1061-4515-8568-e952ed62ca6c-kube-api-access-b4kqv\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.915430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:09.915211    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28d7286-1061-4515-8568-e952ed62ca6c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:10.151665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:10.151583    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"
Oct 30 19:19:10.151851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:10.151585    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp" event={"ID":"d28d7286-1061-4515-8568-e952ed62ca6c","Type":"ContainerDied","Data":"0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e"}
Oct 30 19:19:10.151851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:10.151726    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b54fe25fc5bb444125f8cbb6e36be46006bf226670205cc290372909a0c009e"
Oct 30 19:19:10.414188 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:10.411582    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" podUID="9001629e-1c47-45d5-983c-86fb03663a5d" containerName="payload-processing" containerID="cri-o://4ae2f43fd36be192b34c4d11be282d24d73644d9405a25e05e00defb80429a8e" gracePeriod=25
Oct 30 19:19:11.163567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.163541    5065 generic.go:356] "Generic (PLEG): container finished" podID="9001629e-1c47-45d5-983c-86fb03663a5d" containerID="4ae2f43fd36be192b34c4d11be282d24d73644d9405a25e05e00defb80429a8e" exitCode=0
Oct 30 19:19:11.164010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.163621    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" event={"ID":"9001629e-1c47-45d5-983c-86fb03663a5d","Type":"ContainerDied","Data":"4ae2f43fd36be192b34c4d11be282d24d73644d9405a25e05e00defb80429a8e"}
Oct 30 19:19:11.202495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.202472    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:19:11.324527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.308794    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\") pod \"9001629e-1c47-45d5-983c-86fb03663a5d\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") "
Oct 30 19:19:11.324783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.324418    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume" pod "9001629e-1c47-45d5-983c-86fb03663a5d" (UID: "9001629e-1c47-45d5-983c-86fb03663a5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:11.335320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.335286    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4" pod "9001629e-1c47-45d5-983c-86fb03663a5d" (UID: "9001629e-1c47-45d5-983c-86fb03663a5d"). InnerVolumeSpecName "kube-api-access-mtfx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.324758    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\") pod \"9001629e-1c47-45d5-983c-86fb03663a5d\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") "
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.353186    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\") pod \"9001629e-1c47-45d5-983c-86fb03663a5d\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") "
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.353332    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\") pod \"9001629e-1c47-45d5-983c-86fb03663a5d\" (UID: \"9001629e-1c47-45d5-983c-86fb03663a5d\") "
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.358337    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume" pod "9001629e-1c47-45d5-983c-86fb03663a5d" (UID: "9001629e-1c47-45d5-983c-86fb03663a5d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.359344    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.359372    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfx4\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-kube-api-access-mtfx4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.360798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.359389    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9001629e-1c47-45d5-983c-86fb03663a5d-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.366877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.366832    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials" pod "9001629e-1c47-45d5-983c-86fb03663a5d" (UID: "9001629e-1c47-45d5-983c-86fb03663a5d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.460947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.460921    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9001629e-1c47-45d5-983c-86fb03663a5d-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.803451 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.803419    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"]
Oct 30 19:19:11.804632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804600    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d28d7286-1061-4515-8568-e952ed62ca6c" containerName="revoke-keys"
Oct 30 19:19:11.804632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804627    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28d7286-1061-4515-8568-e952ed62ca6c" containerName="revoke-keys"
Oct 30 19:19:11.804840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804667    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9001629e-1c47-45d5-983c-86fb03663a5d" containerName="payload-processing"
Oct 30 19:19:11.804840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804675    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9001629e-1c47-45d5-983c-86fb03663a5d" containerName="payload-processing"
Oct 30 19:19:11.804959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804919    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28d7286-1061-4515-8568-e952ed62ca6c" containerName="revoke-keys"
Oct 30 19:19:11.804959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.804941    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9001629e-1c47-45d5-983c-86fb03663a5d" containerName="payload-processing"
Oct 30 19:19:11.809045 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.809024    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.840220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.840190    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"]
Oct 30 19:19:11.841552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.841531    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:19:11.844069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.844046    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:11.866721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.866692    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2bj\" (UniqueName: \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.866929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.866904    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.867028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.866988    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.968897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.968869    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2bj\" (UniqueName: \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.969050 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.969022    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.969175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.969071    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.969768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.969745    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.971868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.971842    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:11.997687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:11.997663    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2bj\" (UniqueName: \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:12.123760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.123702    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:12.176055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.176028    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"
Oct 30 19:19:12.176559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.176019    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n" event={"ID":"9001629e-1c47-45d5-983c-86fb03663a5d","Type":"ContainerDied","Data":"b5d3784f781b6b347e070696b495fc9e183d8df0fc251bfe1fea76e06aaff2ba"}
Oct 30 19:19:12.176559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.176166    5065 scope.go:122] "RemoveContainer" containerID="4ae2f43fd36be192b34c4d11be282d24d73644d9405a25e05e00defb80429a8e"
Oct 30 19:19:12.228569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.228532    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:19:12.236779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.236749    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-7c8f8rq6n"]
Oct 30 19:19:12.475786 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:12.475749    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7762e1_3a73_4330_8898_27f71a8298cb.slice/crio-879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe WatchSource:0}: Error finding container 879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe: Status 404 returned error can't find the container with id 879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe
Oct 30 19:19:12.476910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:12.476884    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"]
Oct 30 19:19:13.188599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:13.188504    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl" event={"ID":"dc7762e1-3a73-4330-8898-27f71a8298cb","Type":"ContainerStarted","Data":"070794516a27ad067b385b84204bc8eab83020d962989a44617723602d84a80b"}
Oct 30 19:19:13.188599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:13.188569    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl" event={"ID":"dc7762e1-3a73-4330-8898-27f71a8298cb","Type":"ContainerStarted","Data":"879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe"}
Oct 30 19:19:13.233028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:13.232966    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl" podStartSLOduration=2.232948126 podStartE2EDuration="2.232948126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:13.212051571 +0000 UTC m=+2773.475516546" watchObservedRunningTime="2026-10-30 19:19:13.232948126 +0000 UTC m=+2773.496413073"
Oct 30 19:19:13.894974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:13.894882    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9001629e-1c47-45d5-983c-86fb03663a5d" path="/var/lib/kubelet/pods/9001629e-1c47-45d5-983c-86fb03663a5d/volumes"
Oct 30 19:19:14.202135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:14.201990    5065 generic.go:356] "Generic (PLEG): container finished" podID="dc7762e1-3a73-4330-8898-27f71a8298cb" containerID="070794516a27ad067b385b84204bc8eab83020d962989a44617723602d84a80b" exitCode=0
Oct 30 19:19:14.202135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:14.202103    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl" event={"ID":"dc7762e1-3a73-4330-8898-27f71a8298cb","Type":"ContainerDied","Data":"070794516a27ad067b385b84204bc8eab83020d962989a44617723602d84a80b"}
Oct 30 19:19:15.047999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.047970    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:19:15.076550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.076525    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:19:15.355685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.355639    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"]
Oct 30 19:19:15.372428 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.372393    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.376202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.376160    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"]
Oct 30 19:19:15.402730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.402702    5065 scope.go:122] "RemoveContainer" containerID="1e2466150a5ee99920e7df424b7e5bc546235246f29c3fff840bbf3a2558ee16"
Oct 30 19:19:15.450692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.450603    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.450692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.450666    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55wr\" (UniqueName: \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.451194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.451163    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.458621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.458598    5065 scope.go:122] "RemoveContainer" containerID="cc6c322daab5c350ecc202da2e1c798a288fb42b145b536be3c6d5c9e9a1703b"
Oct 30 19:19:15.459842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.459730    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:15.472362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.472342    5065 scope.go:122] "RemoveContainer" containerID="05fd1bfcbd7d516a9701acae86fbebe7dc9b6c32a647522e899b15f579357e59"
Oct 30 19:19:15.482380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.482365    5065 scope.go:122] "RemoveContainer" containerID="942f88cdd2b489068e5547b5c3c074aafd276e1995ec89009d7c680a637cd2e6"
Oct 30 19:19:15.502616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.502589    5065 scope.go:122] "RemoveContainer" containerID="e42a5a8de1c2f54619ef5c9595303e1bc222680feed4ff0745b27606d0fbb247"
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.552946    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\") pod \"dc7762e1-3a73-4330-8898-27f71a8298cb\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") "
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553069    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\") pod \"dc7762e1-3a73-4330-8898-27f71a8298cb\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") "
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553232    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\" (UniqueName: \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\") pod \"dc7762e1-3a73-4330-8898-27f71a8298cb\" (UID: \"dc7762e1-3a73-4330-8898-27f71a8298cb\") "
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553436    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca" pod "dc7762e1-3a73-4330-8898-27f71a8298cb" (UID: "dc7762e1-3a73-4330-8898-27f71a8298cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553694    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553885    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.553923    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55wr\" (UniqueName: \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.554038    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:15.555645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.554776    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.556922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.556899    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.557585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.557551    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca" pod "dc7762e1-3a73-4330-8898-27f71a8298cb" (UID: "dc7762e1-3a73-4330-8898-27f71a8298cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:15.557707 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.557582    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj" pod "dc7762e1-3a73-4330-8898-27f71a8298cb" (UID: "dc7762e1-3a73-4330-8898-27f71a8298cb"). InnerVolumeSpecName "kube-api-access-bk2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:15.572055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.572030    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55wr\" (UniqueName: \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\") pod \"maas-api-revoke-sub-57f04cb2-tdq9d\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:15.678832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.678787    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk2bj\" (UniqueName: \"kubernetes.io/projected/dc7762e1-3a73-4330-8898-27f71a8298cb-kube-api-access-bk2bj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:15.678832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.678825    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc7762e1-3a73-4330-8898-27f71a8298cb-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:15.694311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:15.694275    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:16.043755 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:16.043660    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c0c782_5018_45e2_adf9_b71d3b7563fc.slice/crio-715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508 WatchSource:0}: Error finding container 715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508: Status 404 returned error can't find the container with id 715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508
Oct 30 19:19:16.045236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:16.045186    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"]
Oct 30 19:19:16.225586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:16.225492    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d" event={"ID":"40c0c782-5018-45e2-adf9-b71d3b7563fc","Type":"ContainerStarted","Data":"715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508"}
Oct 30 19:19:16.227214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:16.227176    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl" event={"ID":"dc7762e1-3a73-4330-8898-27f71a8298cb","Type":"ContainerDied","Data":"879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe"}
Oct 30 19:19:16.227337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:16.227216    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"
Oct 30 19:19:16.227337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:16.227220    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879c5172aebd4a682c48c7307d32fb0e2429590f1e7e5b57f23e99602b57cabe"
Oct 30 19:19:17.239808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:17.239770    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d" event={"ID":"40c0c782-5018-45e2-adf9-b71d3b7563fc","Type":"ContainerStarted","Data":"2ade67e4638b2c036527090f8a34c6b61acc2869c5b721de422deb9a42fcc2e5"}
Oct 30 19:19:17.268363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:17.268315    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d" podStartSLOduration=2.268301416 podStartE2EDuration="2.268301416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:17.262323869 +0000 UTC m=+2777.525788835" watchObservedRunningTime="2026-10-30 19:19:17.268301416 +0000 UTC m=+2777.531766405"
Oct 30 19:19:18.252138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:18.252100    5065 generic.go:356] "Generic (PLEG): container finished" podID="40c0c782-5018-45e2-adf9-b71d3b7563fc" containerID="2ade67e4638b2c036527090f8a34c6b61acc2869c5b721de422deb9a42fcc2e5" exitCode=0
Oct 30 19:19:18.252542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:18.252143    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d" event={"ID":"40c0c782-5018-45e2-adf9-b71d3b7563fc","Type":"ContainerDied","Data":"2ade67e4638b2c036527090f8a34c6b61acc2869c5b721de422deb9a42fcc2e5"}
Oct 30 19:19:19.573728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.573703    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:19.655531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.655469    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\" (UniqueName: \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\") pod \"40c0c782-5018-45e2-adf9-b71d3b7563fc\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") "
Oct 30 19:19:19.655763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.655547    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\") pod \"40c0c782-5018-45e2-adf9-b71d3b7563fc\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") "
Oct 30 19:19:19.655763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.655656    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\") pod \"40c0c782-5018-45e2-adf9-b71d3b7563fc\" (UID: \"40c0c782-5018-45e2-adf9-b71d3b7563fc\") "
Oct 30 19:19:19.656437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.656409    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca" pod "40c0c782-5018-45e2-adf9-b71d3b7563fc" (UID: "40c0c782-5018-45e2-adf9-b71d3b7563fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:19.656784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.656760    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:19.658800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.658772    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca" pod "40c0c782-5018-45e2-adf9-b71d3b7563fc" (UID: "40c0c782-5018-45e2-adf9-b71d3b7563fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:19.659461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.659437    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr" pod "40c0c782-5018-45e2-adf9-b71d3b7563fc" (UID: "40c0c782-5018-45e2-adf9-b71d3b7563fc"). InnerVolumeSpecName "kube-api-access-f55wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:19.757967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.757926    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55wr\" (UniqueName: \"kubernetes.io/projected/40c0c782-5018-45e2-adf9-b71d3b7563fc-kube-api-access-f55wr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:19.757967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:19.757966    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c0c782-5018-45e2-adf9-b71d3b7563fc-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:20.354126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:20.353222    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d" event={"ID":"40c0c782-5018-45e2-adf9-b71d3b7563fc","Type":"ContainerDied","Data":"715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508"}
Oct 30 19:19:20.354126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:20.353287    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715a22e5d4da14ce25588b16ef7f3706cf013e3ab480752325df2e54f299f508"
Oct 30 19:19:20.354126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:20.353356    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"
Oct 30 19:19:20.646243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:20.646157    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"]
Oct 30 19:19:20.654713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:20.654684    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-689c23-j4z2r"]
Oct 30 19:19:21.979883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:21.979832    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13d6882-58e8-4f10-bf47-eb9f3ce9726b" path="/var/lib/kubelet/pods/c13d6882-58e8-4f10-bf47-eb9f3ce9726b/volumes"
Oct 30 19:19:22.046263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:22.046232    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:19:22.138617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:22.138582    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:19:23.059023 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:23.058990    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:19:27.049125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.047386    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing" containerID="cri-o://348a901b7d0d3b50b1d5fd067576bb2783466006c3942d6dd26625816295e3a2" gracePeriod=25
Oct 30 19:19:27.140120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.139308    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing" containerID="cri-o://f41631affb75a2d39a1ec4db2675ed263d19127db9fb20e8fbffd3c688ce523c" gracePeriod=25
Oct 30 19:19:27.546111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.545611    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"]
Oct 30 19:19:27.546922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.546891    5065 cpu_manager.go:414] "containerMap: removing container" podUID="dc7762e1-3a73-4330-8898-27f71a8298cb" containerName="revoke-keys"
Oct 30 19:19:27.546922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.546922    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7762e1-3a73-4330-8898-27f71a8298cb" containerName="revoke-keys"
Oct 30 19:19:27.547148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.546980    5065 cpu_manager.go:414] "containerMap: removing container" podUID="40c0c782-5018-45e2-adf9-b71d3b7563fc" containerName="revoke-keys"
Oct 30 19:19:27.547148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.546987    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c0c782-5018-45e2-adf9-b71d3b7563fc" containerName="revoke-keys"
Oct 30 19:19:27.547260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.547244    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c0c782-5018-45e2-adf9-b71d3b7563fc" containerName="revoke-keys"
Oct 30 19:19:27.547308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.547271    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7762e1-3a73-4330-8898-27f71a8298cb" containerName="revoke-keys"
Oct 30 19:19:27.688111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.657715    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.797167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.758705    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:19:27.797167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.762542    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:27.797167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.793626    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.797167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.793694    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.797167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.793821    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.944264 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.914417    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.969203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.941023    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.988291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.969870    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:27.997110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:27.988597    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:28.093523 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:28.030837    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:28.094055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:28.039035    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\") pod \"maas-api-revoke-sub-288c99c4-dhb7v\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:28.125751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:28.125519    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"]
Oct 30 19:19:28.393052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:28.392950    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:28.708837 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:28.708799    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce5c531_d4c4_4ef7_a317_3014d15e4a3a.slice/crio-91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7 WatchSource:0}: Error finding container 91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7: Status 404 returned error can't find the container with id 91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7
Oct 30 19:19:28.708989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:28.708930    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"]
Oct 30 19:19:29.510161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:29.510123    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v" event={"ID":"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a","Type":"ContainerStarted","Data":"c4961519a6fe0c03952f8fffdec08d59fc6e7b1dfc5db748d6b541f0f601328f"}
Oct 30 19:19:29.510597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:29.510197    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v" event={"ID":"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a","Type":"ContainerStarted","Data":"91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7"}
Oct 30 19:19:29.569403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:29.569347    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v" podStartSLOduration=2.569331833 podStartE2EDuration="2.569331833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:29.567977733 +0000 UTC m=+2789.831442682" watchObservedRunningTime="2026-10-30 19:19:29.569331833 +0000 UTC m=+2789.832796777"
Oct 30 19:19:29.825518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:29.825489    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:19:29.825762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:29.825736    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" containerID="cri-o://a18ea14a1f9f4a3247bd5a9bba52da3ffa8ffbb428f249683a3de23ca6bb356a" gracePeriod=30
Oct 30 19:19:30.328999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.328962    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"]
Oct 30 19:19:30.334246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.334225    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.355632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.355605    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"]
Oct 30 19:19:30.474285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.474249    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbp9\" (UniqueName: \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.474535 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.474513    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.474598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.474564    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.523779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.523743    5065 generic.go:356] "Generic (PLEG): container finished" podID="9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" containerID="c4961519a6fe0c03952f8fffdec08d59fc6e7b1dfc5db748d6b541f0f601328f" exitCode=0
Oct 30 19:19:30.524235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.523836    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v" event={"ID":"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a","Type":"ContainerDied","Data":"c4961519a6fe0c03952f8fffdec08d59fc6e7b1dfc5db748d6b541f0f601328f"}
Oct 30 19:19:30.576682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.576654    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.576832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.576710    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.576884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.576857    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbp9\" (UniqueName: \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.577440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.577417    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.579445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.579381    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.594221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.594195    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbp9\" (UniqueName: \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\") pod \"maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.649443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.649412    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:30.942933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:30.942902    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"]
Oct 30 19:19:31.535491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.535454    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr" event={"ID":"46519fd2-2b1c-42c5-9597-ca1ee6b4625f","Type":"ContainerStarted","Data":"a7060eddec4a389961e67737c4c82d6883b062abf7070204ccfd165cb77fffdf"}
Oct 30 19:19:31.535913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.535516    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr" event={"ID":"46519fd2-2b1c-42c5-9597-ca1ee6b4625f","Type":"ContainerStarted","Data":"4580d7a1b4783bd7cd6116301c9e46e258e8313a03436d7060800026b53c2331"}
Oct 30 19:19:31.563247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.563193    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr" podStartSLOduration=1.563179464 podStartE2EDuration="1.563179464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:31.559114306 +0000 UTC m=+2791.822579263" watchObservedRunningTime="2026-10-30 19:19:31.563179464 +0000 UTC m=+2791.826644426"
Oct 30 19:19:31.727841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.727817    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:31.800260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.800168    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\") pod \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") "
Oct 30 19:19:31.800441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.800423    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\") pod \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") "
Oct 30 19:19:31.800514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.800468    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\") pod \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\" (UID: \"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a\") "
Oct 30 19:19:31.800602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.800554    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca" pod "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" (UID: "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:31.801484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.801455    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:31.803497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.803466    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca" pod "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" (UID: "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:31.806542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.806512    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z" pod "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" (UID: "9ce5c531-d4c4-4ef7-a317-3014d15e4a3a"). InnerVolumeSpecName "kube-api-access-2xd2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:31.902977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.902943    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-kube-api-access-2xd2z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:31.903160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:31.902981    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:32.547065 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:32.547033    5065 generic.go:356] "Generic (PLEG): container finished" podID="46519fd2-2b1c-42c5-9597-ca1ee6b4625f" containerID="a7060eddec4a389961e67737c4c82d6883b062abf7070204ccfd165cb77fffdf" exitCode=0
Oct 30 19:19:32.547535 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:32.547154    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr" event={"ID":"46519fd2-2b1c-42c5-9597-ca1ee6b4625f","Type":"ContainerDied","Data":"a7060eddec4a389961e67737c4c82d6883b062abf7070204ccfd165cb77fffdf"}
Oct 30 19:19:32.548997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:32.548972    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v" event={"ID":"9ce5c531-d4c4-4ef7-a317-3014d15e4a3a","Type":"ContainerDied","Data":"91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7"}
Oct 30 19:19:32.549184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:32.549012    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d45d5267bdc324f5c01f7a72ab72c1d2e2ac03184a2ef71edf6dcbbe6a53f7"
Oct 30 19:19:32.549184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:32.549036    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"
Oct 30 19:19:33.751909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:33.751624    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:33.751909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:33.751642    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:33.995059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:33.994703    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:33.995059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:33.994779    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:34.088209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.082547    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:34.318108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.309242    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\" (UniqueName: \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\") pod \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") "
Oct 30 19:19:34.318108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.309368    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\") pod \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") "
Oct 30 19:19:34.318108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.309623    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\") pod \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\" (UID: \"46519fd2-2b1c-42c5-9597-ca1ee6b4625f\") "
Oct 30 19:19:34.525156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.524115    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca" pod "46519fd2-2b1c-42c5-9597-ca1ee6b4625f" (UID: "46519fd2-2b1c-42c5-9597-ca1ee6b4625f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:34.824317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.821687    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:34.856382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:34.854970    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9" pod "46519fd2-2b1c-42c5-9597-ca1ee6b4625f" (UID: "46519fd2-2b1c-42c5-9597-ca1ee6b4625f"). InnerVolumeSpecName "kube-api-access-6zbp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:35.030370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.030325    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca" pod "46519fd2-2b1c-42c5-9597-ca1ee6b4625f" (UID: "46519fd2-2b1c-42c5-9597-ca1ee6b4625f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:35.043913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.041458    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbp9\" (UniqueName: \"kubernetes.io/projected/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-kube-api-access-6zbp9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.187293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.187197    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46519fd2-2b1c-42c5-9597-ca1ee6b4625f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.263619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.263583    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr" event={"ID":"46519fd2-2b1c-42c5-9597-ca1ee6b4625f","Type":"ContainerDied","Data":"4580d7a1b4783bd7cd6116301c9e46e258e8313a03436d7060800026b53c2331"}
Oct 30 19:19:35.263930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.263907    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4580d7a1b4783bd7cd6116301c9e46e258e8313a03436d7060800026b53c2331"
Oct 30 19:19:35.264152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:35.264135    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"
Oct 30 19:19:36.604885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:36.604847    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:36.825038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:36.824256    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:36.911731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:36.911632    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:19:36.995251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:36.993121    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7" podUID="fa456d21-114b-4b81-93b9-64f94cd6e963" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:36.995251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:36.993721    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:37.212850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.212758    5065 generic.go:356] "Generic (PLEG): container finished" podID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerID="a18ea14a1f9f4a3247bd5a9bba52da3ffa8ffbb428f249683a3de23ca6bb356a" exitCode=0
Oct 30 19:19:37.217326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.217297    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:19:37.266316 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:37.260896    5065 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.366s"
Oct 30 19:19:37.266316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.260937    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" event={"ID":"a7a9e791-1d37-489c-8120-1291be8e20fe","Type":"ContainerDied","Data":"a18ea14a1f9f4a3247bd5a9bba52da3ffa8ffbb428f249683a3de23ca6bb356a"}
Oct 30 19:19:37.266316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.262018    5065 scope.go:122] "RemoveContainer" containerID="a18ea14a1f9f4a3247bd5a9bba52da3ffa8ffbb428f249683a3de23ca6bb356a"
Oct 30 19:19:37.457115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.428282    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.458936 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.458908    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459206    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459253    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459372    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459447    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.459718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.459529    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.460260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.460241    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\") pod \"a7a9e791-1d37-489c-8120-1291be8e20fe\" (UID: \"a7a9e791-1d37-489c-8120-1291be8e20fe\") "
Oct 30 19:19:37.471147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.470760    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:37.471147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.471012    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:37.471147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.471075    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:37.471405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.471232    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:37.474611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.474326    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:37.534714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.534687    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:37.534996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.534972    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:37.535752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.535729    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "kube-api-access-66fxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:37.542252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.542221    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token" pod "a7a9e791-1d37-489c-8120-1291be8e20fe" (UID: "a7a9e791-1d37-489c-8120-1291be8e20fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:37.569035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569011    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569218    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569345    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569447    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569541    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a9e791-1d37-489c-8120-1291be8e20fe-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569660    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569757    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a9e791-1d37-489c-8120-1291be8e20fe-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569843    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:37.569947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:37.569934    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fxd\" (UniqueName: \"kubernetes.io/projected/a7a9e791-1d37-489c-8120-1291be8e20fe-kube-api-access-66fxd\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:38.125135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.125073    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"]
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.126981    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127012    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127054    5065 cpu_manager.go:414] "containerMap: removing container" podUID="46519fd2-2b1c-42c5-9597-ca1ee6b4625f" containerName="revoke-keys"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127064    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46519fd2-2b1c-42c5-9597-ca1ee6b4625f" containerName="revoke-keys"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127127    5065 cpu_manager.go:414] "containerMap: removing container" podUID="9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" containerName="revoke-keys"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127136    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" containerName="revoke-keys"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127437    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" containerName="istio-proxy"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127458    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="46519fd2-2b1c-42c5-9597-ca1ee6b4625f" containerName="revoke-keys"
Oct 30 19:19:38.128124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.127474    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" containerName="revoke-keys"
Oct 30 19:19:38.143108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.137952    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.143691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.143647    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:38.148869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.148848    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:19:38.154464 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.154438    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"]
Oct 30 19:19:38.184140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.183932    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.184687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.184577    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.184923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.184871    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdn4s\" (UniqueName: \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.233712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.233624    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9" event={"ID":"a7a9e791-1d37-489c-8120-1291be8e20fe","Type":"ContainerDied","Data":"7f5d19f6d2fc821b59192d17131e9ed2cac882b4613b08f457e41c19beeba930"}
Oct 30 19:19:38.233712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.233708    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"
Oct 30 19:19:38.263735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.263691    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:19:38.268837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.268810    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-pzkk9"]
Oct 30 19:19:38.286697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.286638    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.286848 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.286820    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdn4s\" (UniqueName: \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.287410 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.287041    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.287713 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.287528    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.292458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.292436    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.301999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.301978    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdn4s\" (UniqueName: \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\") pod \"maas-api-revoke-sub-916dc787-ncqnx\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.469191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.469164    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:38.777994 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:19:38.777962    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953323c3_3705_45d5_8e2e_0d1b6716023b.slice/crio-023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5 WatchSource:0}: Error finding container 023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5: Status 404 returned error can't find the container with id 023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5
Oct 30 19:19:38.780672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:38.780640    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"]
Oct 30 19:19:39.244991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:39.244954    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx" event={"ID":"953323c3-3705-45d5-8e2e-0d1b6716023b","Type":"ContainerStarted","Data":"9c5246bd3374ca8cb8deff0f6dc39b562ee94243f846ba250f40a9bb5216ee37"}
Oct 30 19:19:39.245437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:39.245018    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx" event={"ID":"953323c3-3705-45d5-8e2e-0d1b6716023b","Type":"ContainerStarted","Data":"023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5"}
Oct 30 19:19:39.271209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:39.271166    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx" podStartSLOduration=1.271152824 podStartE2EDuration="1.271152824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:39.268821199 +0000 UTC m=+2799.532286146" watchObservedRunningTime="2026-10-30 19:19:39.271152824 +0000 UTC m=+2799.534617773"
Oct 30 19:19:39.996443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:39.996357    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a9e791-1d37-489c-8120-1291be8e20fe" path="/var/lib/kubelet/pods/a7a9e791-1d37-489c-8120-1291be8e20fe/volumes"
Oct 30 19:19:40.260643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:40.260606    5065 generic.go:356] "Generic (PLEG): container finished" podID="953323c3-3705-45d5-8e2e-0d1b6716023b" containerID="9c5246bd3374ca8cb8deff0f6dc39b562ee94243f846ba250f40a9bb5216ee37" exitCode=0
Oct 30 19:19:40.261043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:40.260743    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx" event={"ID":"953323c3-3705-45d5-8e2e-0d1b6716023b","Type":"ContainerDied","Data":"9c5246bd3374ca8cb8deff0f6dc39b562ee94243f846ba250f40a9bb5216ee37"}
Oct 30 19:19:40.462382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:40.462349    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:19:40.469836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:40.469807    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:19:41.489593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.489563    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:41.589263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.589171    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\" (UniqueName: \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\") pod \"953323c3-3705-45d5-8e2e-0d1b6716023b\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") "
Oct 30 19:19:41.589263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.589256    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\") pod \"953323c3-3705-45d5-8e2e-0d1b6716023b\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") "
Oct 30 19:19:41.589671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.589639    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\") pod \"953323c3-3705-45d5-8e2e-0d1b6716023b\" (UID: \"953323c3-3705-45d5-8e2e-0d1b6716023b\") "
Oct 30 19:19:41.590806 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.590780    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca" pod "953323c3-3705-45d5-8e2e-0d1b6716023b" (UID: "953323c3-3705-45d5-8e2e-0d1b6716023b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:41.593981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.593950    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca" pod "953323c3-3705-45d5-8e2e-0d1b6716023b" (UID: "953323c3-3705-45d5-8e2e-0d1b6716023b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:41.594149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.594112    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s" pod "953323c3-3705-45d5-8e2e-0d1b6716023b" (UID: "953323c3-3705-45d5-8e2e-0d1b6716023b"). InnerVolumeSpecName "kube-api-access-hdn4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:41.695244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.693439    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:41.695579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.695555    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdn4s\" (UniqueName: \"kubernetes.io/projected/953323c3-3705-45d5-8e2e-0d1b6716023b-kube-api-access-hdn4s\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:41.695734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:41.695720    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/953323c3-3705-45d5-8e2e-0d1b6716023b-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:42.281618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:42.281371    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"
Oct 30 19:19:42.281618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:42.281401    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx" event={"ID":"953323c3-3705-45d5-8e2e-0d1b6716023b","Type":"ContainerDied","Data":"023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5"}
Oct 30 19:19:42.281618 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:42.281468    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="023b3629bc9ec41a88ad6ff6a7c367ad4dfaecc94c5bddc5e58ab340d0005ab5"
Oct 30 19:19:43.767109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:43.764280    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:43.773319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:43.773281    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:45.463265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.463011    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" podUID="4850fc60-3a31-40ab-8325-d32f08759e7d" containerName="payload-processing" containerID="cri-o://3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f" gracePeriod=25
Oct 30 19:19:45.471059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.471028    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" podUID="71769a74-18a2-4ad1-9746-dbbcad720f92" containerName="payload-pre-processing" containerID="cri-o://22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac" gracePeriod=25
Oct 30 19:19:45.838858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.838826    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:19:45.876856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.876834    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:19:45.982451 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.982416    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\" (UniqueName: \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\") pod \"71769a74-18a2-4ad1-9746-dbbcad720f92\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") "
Oct 30 19:19:45.982603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.982491    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\" (UniqueName: \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\") pod \"4850fc60-3a31-40ab-8325-d32f08759e7d\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") "
Oct 30 19:19:45.982603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.982552    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\") pod \"71769a74-18a2-4ad1-9746-dbbcad720f92\" (UID: \"71769a74-18a2-4ad1-9746-dbbcad720f92\") "
Oct 30 19:19:45.982751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.982645    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\") pod \"4850fc60-3a31-40ab-8325-d32f08759e7d\" (UID: \"4850fc60-3a31-40ab-8325-d32f08759e7d\") "
Oct 30 19:19:45.983165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.983134    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume" pod "4850fc60-3a31-40ab-8325-d32f08759e7d" (UID: "4850fc60-3a31-40ab-8325-d32f08759e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.983486 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.983149    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume" pod "71769a74-18a2-4ad1-9746-dbbcad720f92" (UID: "71769a74-18a2-4ad1-9746-dbbcad720f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.983654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.983630    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4850fc60-3a31-40ab-8325-d32f08759e7d-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.984924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.984900    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4" pod "4850fc60-3a31-40ab-8325-d32f08759e7d" (UID: "4850fc60-3a31-40ab-8325-d32f08759e7d"). InnerVolumeSpecName "kube-api-access-5v6n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:45.985379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:45.985363    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6" pod "71769a74-18a2-4ad1-9746-dbbcad720f92" (UID: "71769a74-18a2-4ad1-9746-dbbcad720f92"). InnerVolumeSpecName "kube-api-access-l8sk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:46.086456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.086378    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sk6\" (UniqueName: \"kubernetes.io/projected/71769a74-18a2-4ad1-9746-dbbcad720f92-kube-api-access-l8sk6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:46.086456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.086418    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6n4\" (UniqueName: \"kubernetes.io/projected/4850fc60-3a31-40ab-8325-d32f08759e7d-kube-api-access-5v6n4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:46.086456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.086436    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71769a74-18a2-4ad1-9746-dbbcad720f92-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:46.407164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.389775    5065 generic.go:356] "Generic (PLEG): container finished" podID="71769a74-18a2-4ad1-9746-dbbcad720f92" containerID="22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac" exitCode=0
Oct 30 19:19:46.407164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.390120    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" event={"ID":"71769a74-18a2-4ad1-9746-dbbcad720f92","Type":"ContainerDied","Data":"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"}
Oct 30 19:19:46.407164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.390178    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr" event={"ID":"71769a74-18a2-4ad1-9746-dbbcad720f92","Type":"ContainerDied","Data":"8b4834d5982426efff6c82715df4b845b96808f2756d93eb92ffa3c42383c513"}
Oct 30 19:19:46.407164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.390208    5065 scope.go:122] "RemoveContainer" containerID="22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"
Oct 30 19:19:46.407164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.390370    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"
Oct 30 19:19:46.439110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.435396    5065 generic.go:356] "Generic (PLEG): container finished" podID="4850fc60-3a31-40ab-8325-d32f08759e7d" containerID="3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f" exitCode=0
Oct 30 19:19:46.439110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.435455    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" event={"ID":"4850fc60-3a31-40ab-8325-d32f08759e7d","Type":"ContainerDied","Data":"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"}
Oct 30 19:19:46.439110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.435504    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz" event={"ID":"4850fc60-3a31-40ab-8325-d32f08759e7d","Type":"ContainerDied","Data":"c2bd0635db6886850ab2b8b7c640f377a191f68058eea27c20a138cfd37beb31"}
Oct 30 19:19:46.439110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.435586    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"
Oct 30 19:19:46.553681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.552502    5065 scope.go:122] "RemoveContainer" containerID="22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"
Oct 30 19:19:46.570398 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:46.570286    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac\": container with ID starting with 22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac not found: ID does not exist" containerID="22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"
Oct 30 19:19:46.570398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.570346    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac"} err="failed to get container status \"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac\": rpc error: code = NotFound desc = could not find container \"22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac\": container with ID starting with 22de8d4ffd0d405a953b60d601bcb427089548103c3cfa026c45fdf1d4836dac not found: ID does not exist"
Oct 30 19:19:46.570398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.570370    5065 scope.go:122] "RemoveContainer" containerID="3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"
Oct 30 19:19:46.649110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.647996    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:19:46.668190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.665671    5065 scope.go:122] "RemoveContainer" containerID="3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"
Oct 30 19:19:46.706222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.682569    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"]
Oct 30 19:19:46.706222 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:46.690674    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f\": container with ID starting with 3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f not found: ID does not exist" containerID="3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"
Oct 30 19:19:46.706222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.690717    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f"} err="failed to get container status \"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f\": rpc error: code = NotFound desc = could not find container \"3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f\": container with ID starting with 3e997f21cc36b089b4d3f612b41cbea223aef8ca0444bc54ede444eb0e7e8a7f not found: ID does not exist"
Oct 30 19:19:46.706222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.704638    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e77b1d-59cb6c7f66-b4rrz"]
Oct 30 19:19:46.710243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.710170    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8c554-lpkwl"]
Oct 30 19:19:46.716609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.716586    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:19:46.721111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:46.721074    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e77b1d-5d76b5dfdc-tgjfr"]
Oct 30 19:19:47.454771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.454744    5065 generic.go:356] "Generic (PLEG): container finished" podID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerID="f41631affb75a2d39a1ec4db2675ed263d19127db9fb20e8fbffd3c688ce523c" exitCode=0
Oct 30 19:19:47.454907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.454815    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" event={"ID":"0710a88e-b40d-4031-99ee-9ab808e382f5","Type":"ContainerDied","Data":"f41631affb75a2d39a1ec4db2675ed263d19127db9fb20e8fbffd3c688ce523c"}
Oct 30 19:19:47.456756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.456726    5065 generic.go:356] "Generic (PLEG): container finished" podID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerID="348a901b7d0d3b50b1d5fd067576bb2783466006c3942d6dd26625816295e3a2" exitCode=0
Oct 30 19:19:47.456882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.456780    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" event={"ID":"da7d5e5d-dd35-422e-83d8-f97b120a101a","Type":"ContainerDied","Data":"348a901b7d0d3b50b1d5fd067576bb2783466006c3942d6dd26625816295e3a2"}
Oct 30 19:19:47.494234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.490725    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:19:47.566572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.566546    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.593472    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\" (UniqueName: \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\") pod \"da7d5e5d-dd35-422e-83d8-f97b120a101a\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") "
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.593574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\" (UniqueName: \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\") pod \"0710a88e-b40d-4031-99ee-9ab808e382f5\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") "
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.593705    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\") pod \"da7d5e5d-dd35-422e-83d8-f97b120a101a\" (UID: \"da7d5e5d-dd35-422e-83d8-f97b120a101a\") "
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.593833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\") pod \"0710a88e-b40d-4031-99ee-9ab808e382f5\" (UID: \"0710a88e-b40d-4031-99ee-9ab808e382f5\") "
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.594626    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume" pod "da7d5e5d-dd35-422e-83d8-f97b120a101a" (UID: "da7d5e5d-dd35-422e-83d8-f97b120a101a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.595361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume" pod "0710a88e-b40d-4031-99ee-9ab808e382f5" (UID: "0710a88e-b40d-4031-99ee-9ab808e382f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.596025    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7d5e5d-dd35-422e-83d8-f97b120a101a-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:47.598242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.596049    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0710a88e-b40d-4031-99ee-9ab808e382f5-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:47.601693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.601655    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb" pod "da7d5e5d-dd35-422e-83d8-f97b120a101a" (UID: "da7d5e5d-dd35-422e-83d8-f97b120a101a"). InnerVolumeSpecName "kube-api-access-bjgmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:47.603590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.603562    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp" pod "0710a88e-b40d-4031-99ee-9ab808e382f5" (UID: "0710a88e-b40d-4031-99ee-9ab808e382f5"). InnerVolumeSpecName "kube-api-access-7pfpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:47.698026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.697989    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjgmb\" (UniqueName: \"kubernetes.io/projected/da7d5e5d-dd35-422e-83d8-f97b120a101a-kube-api-access-bjgmb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:47.698026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.698022    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfpp\" (UniqueName: \"kubernetes.io/projected/0710a88e-b40d-4031-99ee-9ab808e382f5-kube-api-access-7pfpp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:47.953166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.952851    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4850fc60-3a31-40ab-8325-d32f08759e7d" path="/var/lib/kubelet/pods/4850fc60-3a31-40ab-8325-d32f08759e7d/volumes"
Oct 30 19:19:47.956119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.953686    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71769a74-18a2-4ad1-9746-dbbcad720f92" path="/var/lib/kubelet/pods/71769a74-18a2-4ad1-9746-dbbcad720f92/volumes"
Oct 30 19:19:47.956119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:47.954418    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7762e1-3a73-4330-8898-27f71a8298cb" path="/var/lib/kubelet/pods/dc7762e1-3a73-4330-8898-27f71a8298cb/volumes"
Oct 30 19:19:48.507109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.503693    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6" event={"ID":"0710a88e-b40d-4031-99ee-9ab808e382f5","Type":"ContainerDied","Data":"ac4a77c958de5634189f09a8ab59f495bab8d3f538a04112a8a80d19ba96b1a1"}
Oct 30 19:19:48.507109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.503768    5065 scope.go:122] "RemoveContainer" containerID="f41631affb75a2d39a1ec4db2675ed263d19127db9fb20e8fbffd3c688ce523c"
Oct 30 19:19:48.507109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.503875    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"
Oct 30 19:19:48.549572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.538962    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6" event={"ID":"da7d5e5d-dd35-422e-83d8-f97b120a101a","Type":"ContainerDied","Data":"6f8cc82455f2dc2f244836616f5c632f0b7fbdb90c5f03887f0c2ba020fc6c42"}
Oct 30 19:19:48.549572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.539073    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"
Oct 30 19:19:48.579311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.579193    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:19:48.649123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.647613    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8c554-5b948888ff-mfvc6"]
Oct 30 19:19:48.726298 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.725706    5065 scope.go:122] "RemoveContainer" containerID="348a901b7d0d3b50b1d5fd067576bb2783466006c3942d6dd26625816295e3a2"
Oct 30 19:19:48.736454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.728784    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:19:48.751116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:48.750230    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8c554-68c59d9cd8-msvl6"]
Oct 30 19:19:49.997330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:49.994431    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" path="/var/lib/kubelet/pods/0710a88e-b40d-4031-99ee-9ab808e382f5/volumes"
Oct 30 19:19:50.035109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:50.030164    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" path="/var/lib/kubelet/pods/da7d5e5d-dd35-422e-83d8-f97b120a101a/volumes"
Oct 30 19:19:53.059570 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:19:53.059526    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:19:54.052862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:54.052792    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"]
Oct 30 19:19:54.130825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:54.130786    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e77b1d-sntkr"]
Oct 30 19:19:55.734121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734071    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"]
Oct 30 19:19:55.734904 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734887    5065 cpu_manager.go:414] "containerMap: removing container" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing"
Oct 30 19:19:55.734956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734906    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing"
Oct 30 19:19:55.734956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734921    5065 cpu_manager.go:414] "containerMap: removing container" podUID="71769a74-18a2-4ad1-9746-dbbcad720f92" containerName="payload-pre-processing"
Oct 30 19:19:55.734956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734926    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71769a74-18a2-4ad1-9746-dbbcad720f92" containerName="payload-pre-processing"
Oct 30 19:19:55.734956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734948    5065 cpu_manager.go:414] "containerMap: removing container" podUID="953323c3-3705-45d5-8e2e-0d1b6716023b" containerName="revoke-keys"
Oct 30 19:19:55.734956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734954    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953323c3-3705-45d5-8e2e-0d1b6716023b" containerName="revoke-keys"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734983    5065 cpu_manager.go:414] "containerMap: removing container" podUID="4850fc60-3a31-40ab-8325-d32f08759e7d" containerName="payload-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.734989    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4850fc60-3a31-40ab-8325-d32f08759e7d" containerName="payload-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735001    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735006    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735139    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="4850fc60-3a31-40ab-8325-d32f08759e7d" containerName="payload-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735149    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0710a88e-b40d-4031-99ee-9ab808e382f5" containerName="payload-pre-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735158    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7d5e5d-dd35-422e-83d8-f97b120a101a" containerName="payload-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735165    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="71769a74-18a2-4ad1-9746-dbbcad720f92" containerName="payload-pre-processing"
Oct 30 19:19:55.735233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.735172    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="953323c3-3705-45d5-8e2e-0d1b6716023b" containerName="revoke-keys"
Oct 30 19:19:55.744128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.740835    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.745860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.745834    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"]
Oct 30 19:19:55.746148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.746125    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:55.748043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.748024    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:19:55.830823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.830788    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.831010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.830833    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d5h6\" (UniqueName: \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.831054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.831029    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.898266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.898231    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46519fd2-2b1c-42c5-9597-ca1ee6b4625f" path="/var/lib/kubelet/pods/46519fd2-2b1c-42c5-9597-ca1ee6b4625f/volumes"
Oct 30 19:19:55.933009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.932973    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.933237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.933032    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d5h6\" (UniqueName: \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.933237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.933199    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.933684 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.933656    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.935881 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.935850    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:55.949173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:55.949098    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d5h6\" (UniqueName: \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:56.059895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:56.059871    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:56.394170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:56.392487    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"]
Oct 30 19:19:56.815383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:56.815343    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2" event={"ID":"78e547a7-d5a0-4a07-a71d-ded701c1605c","Type":"ContainerStarted","Data":"5fbd6ee474766d074bce7488e152ebcd2e9b35e709deaec8cf467dfecac5f77b"}
Oct 30 19:19:57.828029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:57.827993    5065 generic.go:356] "Generic (PLEG): container finished" podID="78e547a7-d5a0-4a07-a71d-ded701c1605c" containerID="4bcf30ca91bdb9d1ae70ca42a41b3745dce733257218305cd34e998ce65f6587" exitCode=0
Oct 30 19:19:57.828510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:57.828101    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2" event={"ID":"78e547a7-d5a0-4a07-a71d-ded701c1605c","Type":"ContainerDied","Data":"4bcf30ca91bdb9d1ae70ca42a41b3745dce733257218305cd34e998ce65f6587"}
Oct 30 19:19:58.194781 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.194706    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"]
Oct 30 19:19:58.200887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.200865    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.219619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.219584    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wm9k\" (UniqueName: \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.219790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.219715    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.219867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.219803    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.220844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.220821    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"]
Oct 30 19:19:58.321311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.321272    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wm9k\" (UniqueName: \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.321529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.321327    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.321529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.321374    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.322024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.321986    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.324657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.324632    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.336384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.336355    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wm9k\" (UniqueName: \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\") pod \"maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.533756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.533720    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:19:58.833658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:58.833629    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"]
Oct 30 19:19:59.044230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.044204    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:59.139712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.139620    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\") pod \"78e547a7-d5a0-4a07-a71d-ded701c1605c\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") "
Oct 30 19:19:59.139712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.139658    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\" (UniqueName: \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\") pod \"78e547a7-d5a0-4a07-a71d-ded701c1605c\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") "
Oct 30 19:19:59.139957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.139885    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\") pod \"78e547a7-d5a0-4a07-a71d-ded701c1605c\" (UID: \"78e547a7-d5a0-4a07-a71d-ded701c1605c\") "
Oct 30 19:19:59.140407 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.140368    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca" pod "78e547a7-d5a0-4a07-a71d-ded701c1605c" (UID: "78e547a7-d5a0-4a07-a71d-ded701c1605c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:59.140756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.140733    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.142047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.142022    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6" pod "78e547a7-d5a0-4a07-a71d-ded701c1605c" (UID: "78e547a7-d5a0-4a07-a71d-ded701c1605c"). InnerVolumeSpecName "kube-api-access-4d5h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:59.142158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.142064    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca" pod "78e547a7-d5a0-4a07-a71d-ded701c1605c" (UID: "78e547a7-d5a0-4a07-a71d-ded701c1605c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:59.242488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.242460    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e547a7-d5a0-4a07-a71d-ded701c1605c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.242488 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.242488    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d5h6\" (UniqueName: \"kubernetes.io/projected/78e547a7-d5a0-4a07-a71d-ded701c1605c-kube-api-access-4d5h6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:59.858756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.858722    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s" event={"ID":"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1","Type":"ContainerStarted","Data":"6dc37c8c4ebb35247c63ca5dc06a5a4be3db223d80737f5b5edf267dfd4335bf"}
Oct 30 19:19:59.859230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.858788    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s" event={"ID":"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1","Type":"ContainerStarted","Data":"0171dabafe40036b50a5d680630d3b4ae0026225ff2ee1ade3c9b38c55cd5f48"}
Oct 30 19:19:59.860468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.860445    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"
Oct 30 19:19:59.860596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.860494    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2" event={"ID":"78e547a7-d5a0-4a07-a71d-ded701c1605c","Type":"ContainerDied","Data":"5fbd6ee474766d074bce7488e152ebcd2e9b35e709deaec8cf467dfecac5f77b"}
Oct 30 19:19:59.860596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.860549    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbd6ee474766d074bce7488e152ebcd2e9b35e709deaec8cf467dfecac5f77b"
Oct 30 19:19:59.885639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:19:59.885591    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s" podStartSLOduration=1.8855755680000001 podStartE2EDuration="1.885575568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:59.881603981 +0000 UTC m=+2820.145068928" watchObservedRunningTime="2026-10-30 19:19:59.885575568 +0000 UTC m=+2820.149040517"
Oct 30 19:20:00.872647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:00.872602    5065 generic.go:356] "Generic (PLEG): container finished" podID="3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" containerID="6dc37c8c4ebb35247c63ca5dc06a5a4be3db223d80737f5b5edf267dfd4335bf" exitCode=0
Oct 30 19:20:00.873185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:00.872686    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s" event={"ID":"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1","Type":"ContainerDied","Data":"6dc37c8c4ebb35247c63ca5dc06a5a4be3db223d80737f5b5edf267dfd4335bf"}
Oct 30 19:20:02.043958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.043913    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:20:02.044645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.044611    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" containerID="cri-o://4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9" gracePeriod=30
Oct 30 19:20:02.500740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.491360    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:20:02.618118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.617998    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\") pod \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") "
Oct 30 19:20:02.618320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.618119    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\") pod \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") "
Oct 30 19:20:02.618320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.618184    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\" (UniqueName: \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\") pod \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\" (UID: \"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1\") "
Oct 30 19:20:02.618490 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.618465    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca" pod "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" (UID: "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:02.619312 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.619278    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:02.620657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.620619    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca" pod "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" (UID: "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:02.620941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.620909    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k" pod "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" (UID: "3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1"). InnerVolumeSpecName "kube-api-access-6wm9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:02.648106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.648057    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"]
Oct 30 19:20:02.648939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.648921    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" containerName="revoke-keys"
Oct 30 19:20:02.648985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.648942    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" containerName="revoke-keys"
Oct 30 19:20:02.648985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.648961    5065 cpu_manager.go:414] "containerMap: removing container" podUID="78e547a7-d5a0-4a07-a71d-ded701c1605c" containerName="revoke-keys"
Oct 30 19:20:02.648985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.648965    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e547a7-d5a0-4a07-a71d-ded701c1605c" containerName="revoke-keys"
Oct 30 19:20:02.649130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.649119    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" containerName="revoke-keys"
Oct 30 19:20:02.649175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.649135    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e547a7-d5a0-4a07-a71d-ded701c1605c" containerName="revoke-keys"
Oct 30 19:20:02.653402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.653378    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.701394 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.701286    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"]
Oct 30 19:20:02.727753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.727719    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:02.727753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.727750    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wm9k\" (UniqueName: \"kubernetes.io/projected/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1-kube-api-access-6wm9k\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:02.829423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.829385    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.829689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.829651    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.829854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.829838    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnw2\" (UniqueName: \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.926032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.925946    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"
Oct 30 19:20:02.926292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.925955    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s" event={"ID":"3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1","Type":"ContainerDied","Data":"0171dabafe40036b50a5d680630d3b4ae0026225ff2ee1ade3c9b38c55cd5f48"}
Oct 30 19:20:02.926292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.926273    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0171dabafe40036b50a5d680630d3b4ae0026225ff2ee1ade3c9b38c55cd5f48"
Oct 30 19:20:02.931141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.931099    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnw2\" (UniqueName: \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.931344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.931268    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.932028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.932001    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.933037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.933006    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.934587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.934564    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.946736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.946697    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnw2\" (UniqueName: \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:02.970372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:02.970336    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:03.277145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.275266    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:03.277145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.275341    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:03.315816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.315778    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"]
Oct 30 19:20:03.939469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.939348    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl" event={"ID":"11763bf3-9d16-4597-872e-a50bd2a2e796","Type":"ContainerStarted","Data":"6b1cd94a9cee2b77da4919e024dd43db02388a3e22635a223250346e0bc4814b"}
Oct 30 19:20:03.939469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.939435    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl" event={"ID":"11763bf3-9d16-4597-872e-a50bd2a2e796","Type":"ContainerStarted","Data":"5e3b5c0a626900e78174a32d1d0e9a3c64883dc3a269388765cbf0ff4871210d"}
Oct 30 19:20:03.964891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:03.964825    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl" podStartSLOduration=1.9648026459999999 podStartE2EDuration="1.964802646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:03.960804523 +0000 UTC m=+2824.224269486" watchObservedRunningTime="2026-10-30 19:20:03.964802646 +0000 UTC m=+2824.228267594"
Oct 30 19:20:04.953290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:04.953253    5065 generic.go:356] "Generic (PLEG): container finished" podID="11763bf3-9d16-4597-872e-a50bd2a2e796" containerID="6b1cd94a9cee2b77da4919e024dd43db02388a3e22635a223250346e0bc4814b" exitCode=0
Oct 30 19:20:04.953700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:04.953337    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl" event={"ID":"11763bf3-9d16-4597-872e-a50bd2a2e796","Type":"ContainerDied","Data":"6b1cd94a9cee2b77da4919e024dd43db02388a3e22635a223250346e0bc4814b"}
Oct 30 19:20:06.544029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.544002    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:06.654942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.654906    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\") pod \"11763bf3-9d16-4597-872e-a50bd2a2e796\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") "
Oct 30 19:20:06.655211 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.655184    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\") pod \"11763bf3-9d16-4597-872e-a50bd2a2e796\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") "
Oct 30 19:20:06.655346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.655284    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca" pod "11763bf3-9d16-4597-872e-a50bd2a2e796" (UID: "11763bf3-9d16-4597-872e-a50bd2a2e796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:06.655346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.655309    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\" (UniqueName: \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\") pod \"11763bf3-9d16-4597-872e-a50bd2a2e796\" (UID: \"11763bf3-9d16-4597-872e-a50bd2a2e796\") "
Oct 30 19:20:06.656022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.655993    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:06.657491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.657467    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca" pod "11763bf3-9d16-4597-872e-a50bd2a2e796" (UID: "11763bf3-9d16-4597-872e-a50bd2a2e796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:06.657491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.657482    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2" pod "11763bf3-9d16-4597-872e-a50bd2a2e796" (UID: "11763bf3-9d16-4597-872e-a50bd2a2e796"). InnerVolumeSpecName "kube-api-access-xfnw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:06.758027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.757954    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11763bf3-9d16-4597-872e-a50bd2a2e796-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:06.758027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.757988    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnw2\" (UniqueName: \"kubernetes.io/projected/11763bf3-9d16-4597-872e-a50bd2a2e796-kube-api-access-xfnw2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:06.988283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.988254    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"
Oct 30 19:20:06.988283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.988266    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl" event={"ID":"11763bf3-9d16-4597-872e-a50bd2a2e796","Type":"ContainerDied","Data":"5e3b5c0a626900e78174a32d1d0e9a3c64883dc3a269388765cbf0ff4871210d"}
Oct 30 19:20:06.988516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:06.988314    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e3b5c0a626900e78174a32d1d0e9a3c64883dc3a269388765cbf0ff4871210d"
Oct 30 19:20:07.359810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.359780    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:20:07.475911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.475876    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476007    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476208    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476261    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476329    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476403    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:07.476470 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476460    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476547    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476553    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.476624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476602    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476638    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\") pod \"e10d8444-b860-4111-8190-718915129e0e\" (UID: \"e10d8444-b860-4111-8190-718915129e0e\") "
Oct 30 19:20:07.476820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476694    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.476925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.476899    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.477168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477076    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.477551 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477511    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.477551 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477538    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.478061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477554    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.478061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477568    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e10d8444-b860-4111-8190-718915129e0e-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.478061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.477582    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.479928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.479906    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:07.480013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.479969    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:07.480127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.480039    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.480757 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.480734    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc" pod "e10d8444-b860-4111-8190-718915129e0e" (UID: "e10d8444-b860-4111-8190-718915129e0e"). InnerVolumeSpecName "kube-api-access-vgsgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:07.579808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.579771    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e10d8444-b860-4111-8190-718915129e0e-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.579808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.579811    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.580336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.579828    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsgc\" (UniqueName: \"kubernetes.io/projected/e10d8444-b860-4111-8190-718915129e0e-kube-api-access-vgsgc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.580336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:07.579843    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e10d8444-b860-4111-8190-718915129e0e-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:08.003820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.003778    5065 generic.go:356] "Generic (PLEG): container finished" podID="e10d8444-b860-4111-8190-718915129e0e" containerID="4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9" exitCode=0
Oct 30 19:20:08.004191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.003844    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerDied","Data":"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"}
Oct 30 19:20:08.004191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.003919    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4" event={"ID":"e10d8444-b860-4111-8190-718915129e0e","Type":"ContainerDied","Data":"10537d82624151f844a59222b70cd829b3eac4d02a32cda2beb566d066f28943"}
Oct 30 19:20:08.004191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.003856    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"
Oct 30 19:20:08.004191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.003950    5065 scope.go:122] "RemoveContainer" containerID="4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"
Oct 30 19:20:08.024802 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.024783    5065 scope.go:122] "RemoveContainer" containerID="8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"
Oct 30 19:20:08.037972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.037918    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:20:08.040952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.040933    5065 scope.go:122] "RemoveContainer" containerID="4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"
Oct 30 19:20:08.041358 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:08.041332    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9\": container with ID starting with 4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9 not found: ID does not exist" containerID="4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"
Oct 30 19:20:08.041442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.041371    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9"} err="failed to get container status \"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9\": rpc error: code = NotFound desc = could not find container \"4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9\": container with ID starting with 4f11e678fa41d77a6e1e4202e89eaf8df4a3ed136296642c844f20faf7ab28b9 not found: ID does not exist"
Oct 30 19:20:08.041442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.041398    5065 scope.go:122] "RemoveContainer" containerID="8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"
Oct 30 19:20:08.041741 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:08.041721    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a\": container with ID starting with 8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a not found: ID does not exist" containerID="8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"
Oct 30 19:20:08.041830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.041748    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a"} err="failed to get container status \"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a\": rpc error: code = NotFound desc = could not find container \"8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a\": container with ID starting with 8f57e54f334cf7dd122b939a5515e196ead1704ff88b9d49c093e3fd9265635a not found: ID does not exist"
Oct 30 19:20:08.044278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.044255    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-7t5s4"]
Oct 30 19:20:08.549362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.549321    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:20:08.576108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:08.576055    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:20:09.894190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:09.894154    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10d8444-b860-4111-8190-718915129e0e" path="/var/lib/kubelet/pods/e10d8444-b860-4111-8190-718915129e0e/volumes"
Oct 30 19:20:12.890679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:12.890601    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 19:20:12.907681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:12.907648    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 19:20:13.550367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:13.550068    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing" containerID="cri-o://c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092" gracePeriod=25
Oct 30 19:20:13.581244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:13.581198    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing" containerID="cri-o://fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc" gracePeriod=25
Oct 30 19:20:15.594099 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:15.594061    5065 scope.go:122] "RemoveContainer" containerID="2f6cfb6aac5c971c44c2c4009587e6e43407a2542e8db28d2b9f0681225c68d4"
Oct 30 19:20:15.606327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:15.606299    5065 scope.go:122] "RemoveContainer" containerID="9eb3e7f30b34288425af33deaf9769f3175e94c2536f61e0e4a7aa9734766812"
Oct 30 19:20:15.618235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:15.618217    5065 scope.go:122] "RemoveContainer" containerID="cad8a78d92b72b7f6d1f2d3f9d01d5e89fc754ad41b50a42601fb285e5f393d1"
Oct 30 19:20:17.292436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.283280    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"]
Oct 30 19:20:17.292436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.291570    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e94aa-trlf2"]
Oct 30 19:20:17.408142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.408103    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:17.408142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.408122    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:17.891161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.891118    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" podUID="e019ae70-ddae-41c3-94ad-24364cfd4675" containerName="payload-processing" containerID="cri-o://e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a" gracePeriod=25
Oct 30 19:20:17.895858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.895818    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e547a7-d5a0-4a07-a71d-ded701c1605c" path="/var/lib/kubelet/pods/78e547a7-d5a0-4a07-a71d-ded701c1605c/volumes"
Oct 30 19:20:17.908942 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:17.908908    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" podUID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" containerName="payload-pre-processing" containerID="cri-o://2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185" gracePeriod=25
Oct 30 19:20:18.251320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.251292    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 19:20:18.260439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.260410    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\" (UniqueName: \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\") pod \"e019ae70-ddae-41c3-94ad-24364cfd4675\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") "
Oct 30 19:20:18.260819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.260800    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\") pod \"e019ae70-ddae-41c3-94ad-24364cfd4675\" (UID: \"e019ae70-ddae-41c3-94ad-24364cfd4675\") "
Oct 30 19:20:18.261194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.261170    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume" pod "e019ae70-ddae-41c3-94ad-24364cfd4675" (UID: "e019ae70-ddae-41c3-94ad-24364cfd4675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:18.261473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.261458    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e019ae70-ddae-41c3-94ad-24364cfd4675-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:18.262640 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.262623    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj" pod "e019ae70-ddae-41c3-94ad-24364cfd4675" (UID: "e019ae70-ddae-41c3-94ad-24364cfd4675"). InnerVolumeSpecName "kube-api-access-pxwnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:18.291076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.291050    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 19:20:18.330838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.330803    5065 generic.go:356] "Generic (PLEG): container finished" podID="e019ae70-ddae-41c3-94ad-24364cfd4675" containerID="e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a" exitCode=0
Oct 30 19:20:18.331346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.330858    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"
Oct 30 19:20:18.331346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.330886    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" event={"ID":"e019ae70-ddae-41c3-94ad-24364cfd4675","Type":"ContainerDied","Data":"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"}
Oct 30 19:20:18.331346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.330942    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48" event={"ID":"e019ae70-ddae-41c3-94ad-24364cfd4675","Type":"ContainerDied","Data":"d2e45eb5e126bdddd3604db3b234fc5f39626ae6c0ad044ea8bbd5e006bd164c"}
Oct 30 19:20:18.331346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.330973    5065 scope.go:122] "RemoveContainer" containerID="e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"
Oct 30 19:20:18.333146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.333120    5065 generic.go:356] "Generic (PLEG): container finished" podID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" containerID="2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185" exitCode=0
Oct 30 19:20:18.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.333164    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"
Oct 30 19:20:18.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.333202    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" event={"ID":"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c","Type":"ContainerDied","Data":"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"}
Oct 30 19:20:18.333288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.333242    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj" event={"ID":"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c","Type":"ContainerDied","Data":"7fdc20175754255a45a0d09ebfa95690fec602a6142aec1c86d365228acdccf8"}
Oct 30 19:20:18.346292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.346263    5065 scope.go:122] "RemoveContainer" containerID="e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"
Oct 30 19:20:18.346597 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:18.346577    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a\": container with ID starting with e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a not found: ID does not exist" containerID="e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"
Oct 30 19:20:18.346712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.346603    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a"} err="failed to get container status \"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a\": rpc error: code = NotFound desc = could not find container \"e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a\": container with ID starting with e3e15c04712703872fa3b46c16f2df7092a88005813cda62080beb15db8ee59a not found: ID does not exist"
Oct 30 19:20:18.346712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.346619    5065 scope.go:122] "RemoveContainer" containerID="2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"
Oct 30 19:20:18.358662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.358644    5065 scope.go:122] "RemoveContainer" containerID="2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"
Oct 30 19:20:18.359004 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:18.358977    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185\": container with ID starting with 2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185 not found: ID does not exist" containerID="2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"
Oct 30 19:20:18.359141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.359014    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185"} err="failed to get container status \"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185\": rpc error: code = NotFound desc = could not find container \"2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185\": container with ID starting with 2e01cbcbb0e3ef5faeacc3639c2bf0302fb7a228cc634d35653cb77dcf73d185 not found: ID does not exist"
Oct 30 19:20:18.362013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.361989    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\") pod \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") "
Oct 30 19:20:18.362295 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.362277    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\" (UniqueName: \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\") pod \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\" (UID: \"7c58cd20-bd59-4bd2-b135-7bfbf5eb008c\") "
Oct 30 19:20:18.362420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.362401    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume" pod "7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" (UID: "7c58cd20-bd59-4bd2-b135-7bfbf5eb008c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:18.363066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.363002    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwnj\" (UniqueName: \"kubernetes.io/projected/e019ae70-ddae-41c3-94ad-24364cfd4675-kube-api-access-pxwnj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:18.363066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.363028    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:18.364731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.364702    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7" pod "7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" (UID: "7c58cd20-bd59-4bd2-b135-7bfbf5eb008c"). InnerVolumeSpecName "kube-api-access-5fkx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:18.367582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.367559    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 19:20:18.373600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.373581    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c3a5b9-bd698cc76-pmb48"]
Oct 30 19:20:18.493172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.487733    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fkx7\" (UniqueName: \"kubernetes.io/projected/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c-kube-api-access-5fkx7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:18.662225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.662194    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 19:20:18.666985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:18.666967    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c3a5b9-765c45f66c-4lntj"]
Oct 30 19:20:19.892526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:19.892490    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" path="/var/lib/kubelet/pods/7c58cd20-bd59-4bd2-b135-7bfbf5eb008c/volumes"
Oct 30 19:20:19.893044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:19.893027    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e019ae70-ddae-41c3-94ad-24364cfd4675" path="/var/lib/kubelet/pods/e019ae70-ddae-41c3-94ad-24364cfd4675/volumes"
Oct 30 19:20:23.141543 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:23.141508    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:20:24.153826 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:24.153790    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"]
Oct 30 19:20:24.158700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:24.158675    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-de4bda-nlq4s"]
Oct 30 19:20:24.703945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:24.703915    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"]
Oct 30 19:20:24.711657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:24.711625    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c3a5b9-965sl"]
Oct 30 19:20:25.635742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:25.635697    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:20:25.636259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:25.635917    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy" containerID="cri-o://b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5" gracePeriod=30
Oct 30 19:20:26.061221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.061164    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11763bf3-9d16-4597-872e-a50bd2a2e796" path="/var/lib/kubelet/pods/11763bf3-9d16-4597-872e-a50bd2a2e796/volumes"
Oct 30 19:20:26.063755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.063725    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1" path="/var/lib/kubelet/pods/3dee14d1-0f7e-4fd5-88ad-a3b2397e39d1/volumes"
Oct 30 19:20:26.179074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.179039    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"]
Oct 30 19:20:26.181386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.181357    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e019ae70-ddae-41c3-94ad-24364cfd4675" containerName="payload-processing"
Oct 30 19:20:26.181582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.181568    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e019ae70-ddae-41c3-94ad-24364cfd4675" containerName="payload-processing"
Oct 30 19:20:26.181760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.181748    5065 cpu_manager.go:414] "containerMap: removing container" podUID="11763bf3-9d16-4597-872e-a50bd2a2e796" containerName="revoke-keys"
Oct 30 19:20:26.181856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.181845    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11763bf3-9d16-4597-872e-a50bd2a2e796" containerName="revoke-keys"
Oct 30 19:20:26.181970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.181960    5065 cpu_manager.go:414] "containerMap: removing container" podUID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" containerName="payload-pre-processing"
Oct 30 19:20:26.182103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182072    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" containerName="payload-pre-processing"
Oct 30 19:20:26.182277 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182265    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.182365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182355    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.182450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182441    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.182555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182544    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.183020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.182996    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e019ae70-ddae-41c3-94ad-24364cfd4675" containerName="payload-processing"
Oct 30 19:20:26.183214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.183038    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="11763bf3-9d16-4597-872e-a50bd2a2e796" containerName="revoke-keys"
Oct 30 19:20:26.183214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.183056    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c58cd20-bd59-4bd2-b135-7bfbf5eb008c" containerName="payload-pre-processing"
Oct 30 19:20:26.183214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.183101    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.183214 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.183121    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10d8444-b860-4111-8190-718915129e0e" containerName="istio-proxy"
Oct 30 19:20:26.195421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.195398    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.207245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.207218    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"]
Oct 30 19:20:26.209063 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.209038    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:20:26.210229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.210176    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:26.323289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.323210    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmdj\" (UniqueName: \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.323289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.323284    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.323555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.323529    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.424889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.424850    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmdj\" (UniqueName: \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.425056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.424924    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.425152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.425124    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.425887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.425858    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.427676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.427647    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.438639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.438617    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmdj\" (UniqueName: \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.525982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.525955    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:26.817533 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:20:26.817461    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2641bd26_6a09_474d_90d3_e8a6403a651c.slice/crio-163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af WatchSource:0}: Error finding container 163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af: Status 404 returned error can't find the container with id 163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af
Oct 30 19:20:26.818664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:26.818639    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"]
Oct 30 19:20:27.409299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:27.409259    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:27.409533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:27.409278    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:27.453738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:27.453654    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l" event={"ID":"2641bd26-6a09-474d-90d3-e8a6403a651c","Type":"ContainerStarted","Data":"91b0d7496a08ffc51112b92bcf64b3c160094589e3461fefa8581336a2c4c88d"}
Oct 30 19:20:27.453738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:27.453719    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l" event={"ID":"2641bd26-6a09-474d-90d3-e8a6403a651c","Type":"ContainerStarted","Data":"163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af"}
Oct 30 19:20:27.478483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:27.478436    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l" podStartSLOduration=1.47842243 podStartE2EDuration="1.47842243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:27.474028231 +0000 UTC m=+2847.737493179" watchObservedRunningTime="2026-10-30 19:20:27.47842243 +0000 UTC m=+2847.741887420"
Oct 30 19:20:28.465202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:28.465168    5065 generic.go:356] "Generic (PLEG): container finished" podID="2641bd26-6a09-474d-90d3-e8a6403a651c" containerID="91b0d7496a08ffc51112b92bcf64b3c160094589e3461fefa8581336a2c4c88d" exitCode=0
Oct 30 19:20:28.465577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:28.465247    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l" event={"ID":"2641bd26-6a09-474d-90d3-e8a6403a651c","Type":"ContainerDied","Data":"91b0d7496a08ffc51112b92bcf64b3c160094589e3461fefa8581336a2c4c88d"}
Oct 30 19:20:29.723536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.723506    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:29.885282 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.885202    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\" (UniqueName: \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\") pod \"2641bd26-6a09-474d-90d3-e8a6403a651c\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") "
Oct 30 19:20:29.885524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.885503    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\") pod \"2641bd26-6a09-474d-90d3-e8a6403a651c\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") "
Oct 30 19:20:29.885633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.885593    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\") pod \"2641bd26-6a09-474d-90d3-e8a6403a651c\" (UID: \"2641bd26-6a09-474d-90d3-e8a6403a651c\") "
Oct 30 19:20:29.886531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.886495    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca" pod "2641bd26-6a09-474d-90d3-e8a6403a651c" (UID: "2641bd26-6a09-474d-90d3-e8a6403a651c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:29.888311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.888205    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj" pod "2641bd26-6a09-474d-90d3-e8a6403a651c" (UID: "2641bd26-6a09-474d-90d3-e8a6403a651c"). InnerVolumeSpecName "kube-api-access-rqmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:29.888556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.888532    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca" pod "2641bd26-6a09-474d-90d3-e8a6403a651c" (UID: "2641bd26-6a09-474d-90d3-e8a6403a651c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:29.988253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.988228    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmdj\" (UniqueName: \"kubernetes.io/projected/2641bd26-6a09-474d-90d3-e8a6403a651c-kube-api-access-rqmdj\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.988377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.988274    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:29.988377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:29.988285    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641bd26-6a09-474d-90d3-e8a6403a651c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:30.591667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:30.591525    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l" event={"ID":"2641bd26-6a09-474d-90d3-e8a6403a651c","Type":"ContainerDied","Data":"163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af"}
Oct 30 19:20:30.591902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:30.591693    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"
Oct 30 19:20:30.591902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:30.591782    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163c8267fa0072f0b7b44c1aa1e0e89e25d4e9bbe1ae4d226ac169c8423472af"
Oct 30 19:20:30.972912 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:30.972890    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:20:31.007845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.007819    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.008012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.007889    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.008012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.007957    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.010460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.010434    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.010833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.010801    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "kube-api-access-bzwd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.010964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.010915    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:31.109599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109566    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.109765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109624    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.109765 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109652    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.109907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109806    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.109907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109896    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.110026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109927    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.110026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109948    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.110026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.109982    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\") pod \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\" (UID: \"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5\") "
Oct 30 19:20:31.110213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110080    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.110296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110258    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.110296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110282    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.110586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110571    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110590    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110600    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwd4\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-kube-api-access-bzwd4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110610    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110618    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110626    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110636    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.110758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.110643    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.112011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.111988    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token" pod "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" (UID: "dcf139ad-7550-4b83-a8ff-4c8cc95a96b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.212170 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.212073    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.607293 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.607257    5065 generic.go:356] "Generic (PLEG): container finished" podID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerID="b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5" exitCode=0
Oct 30 19:20:31.607512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.607308    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"
Oct 30 19:20:31.607512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.607340    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" event={"ID":"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5","Type":"ContainerDied","Data":"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"}
Oct 30 19:20:31.607512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.607423    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2" event={"ID":"dcf139ad-7550-4b83-a8ff-4c8cc95a96b5","Type":"ContainerDied","Data":"e3ace449c60f9dc0c57cfd1c9636fa5d9c4d88a95957c2b9b3248f00097da838"}
Oct 30 19:20:31.607512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.607454    5065 scope.go:122] "RemoveContainer" containerID="b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"
Oct 30 19:20:31.623020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.622999    5065 scope.go:122] "RemoveContainer" containerID="b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"
Oct 30 19:20:31.623344 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:31.623323    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5\": container with ID starting with b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5 not found: ID does not exist" containerID="b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"
Oct 30 19:20:31.623477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.623354    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5"} err="failed to get container status \"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5\": rpc error: code = NotFound desc = could not find container \"b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5\": container with ID starting with b84fa301d84ffe365430a007716b4abbbd64fedbfa921a33544640cac58895e5 not found: ID does not exist"
Oct 30 19:20:31.638692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.638671    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:20:31.646307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.646289    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-jjbz2"]
Oct 30 19:20:31.893346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:31.893275    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" path="/var/lib/kubelet/pods/dcf139ad-7550-4b83-a8ff-4c8cc95a96b5/volumes"
Oct 30 19:20:33.697783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.697745    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"]
Oct 30 19:20:33.698619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698598    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2641bd26-6a09-474d-90d3-e8a6403a651c" containerName="revoke-keys"
Oct 30 19:20:33.698619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698619    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2641bd26-6a09-474d-90d3-e8a6403a651c" containerName="revoke-keys"
Oct 30 19:20:33.698818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698656    5065 cpu_manager.go:414] "containerMap: removing container" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy"
Oct 30 19:20:33.698818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698661    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy"
Oct 30 19:20:33.698818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698811    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf139ad-7550-4b83-a8ff-4c8cc95a96b5" containerName="istio-proxy"
Oct 30 19:20:33.698990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.698834    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2641bd26-6a09-474d-90d3-e8a6403a651c" containerName="revoke-keys"
Oct 30 19:20:33.702827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.702807    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:33.712187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.711950    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:33.716572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.716469    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"]
Oct 30 19:20:33.716572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.716547    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:20:33.857272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.857236    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:20:33.857565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.857534    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" containerID="cri-o://83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de" gracePeriod=30
Oct 30 19:20:33.882582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.879407    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:33.882582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.879534    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:33.882582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:33.879753    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5gvz\" (UniqueName: \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.073133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.039720    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.073133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.039857    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.073133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.040118    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gvz\" (UniqueName: \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.080738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.080350    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.250889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.250647    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.257736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.257707    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gvz\" (UniqueName: \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.347112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.340384    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:20:34.368145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.366669    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:34.410317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.410291    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"]
Oct 30 19:20:34.422155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.415640    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing"
Oct 30 19:20:34.422850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.422829    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing"
Oct 30 19:20:34.423363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.423342    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerName="payload-processing"
Oct 30 19:20:34.429411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.429389    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"]
Oct 30 19:20:34.429648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.429634    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.471471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.466292    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:20:34.489321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.486337    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\") pod \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") "
Oct 30 19:20:34.489321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.486633    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume" pod "f41130fe-bf8b-4ac8-8fd2-75bd98b68832" (UID: "f41130fe-bf8b-4ac8-8fd2-75bd98b68832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:34.489321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.486693    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\" (UniqueName: \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\") pod \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\" (UID: \"f41130fe-bf8b-4ac8-8fd2-75bd98b68832\") "
Oct 30 19:20:34.489321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.487468    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:34.495575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.495544    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx" pod "f41130fe-bf8b-4ac8-8fd2-75bd98b68832" (UID: "f41130fe-bf8b-4ac8-8fd2-75bd98b68832"). InnerVolumeSpecName "kube-api-access-fvwbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:34.588514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.588486    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\") pod \"8969b3a8-67af-408b-a1a5-d3657496c3d7\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") "
Oct 30 19:20:34.588675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.588597    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\" (UniqueName: \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\") pod \"8969b3a8-67af-408b-a1a5-d3657496c3d7\" (UID: \"8969b3a8-67af-408b-a1a5-d3657496c3d7\") "
Oct 30 19:20:34.588927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.588903    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume" pod "8969b3a8-67af-408b-a1a5-d3657496c3d7" (UID: "8969b3a8-67af-408b-a1a5-d3657496c3d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:34.589016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.588995    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfh8l\" (UniqueName: \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.589067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.589048    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.589476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.589452    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.589674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.589657    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969b3a8-67af-408b-a1a5-d3657496c3d7-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:34.589717 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.589681    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwbx\" (UniqueName: \"kubernetes.io/projected/f41130fe-bf8b-4ac8-8fd2-75bd98b68832-kube-api-access-fvwbx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:34.590913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.590884    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w" pod "8969b3a8-67af-408b-a1a5-d3657496c3d7" (UID: "8969b3a8-67af-408b-a1a5-d3657496c3d7"). InnerVolumeSpecName "kube-api-access-nlv8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:34.641700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.641668    5065 generic.go:356] "Generic (PLEG): container finished" podID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerID="fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc" exitCode=0
Oct 30 19:20:34.641877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.641727    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"
Oct 30 19:20:34.641877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.641769    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" event={"ID":"8969b3a8-67af-408b-a1a5-d3657496c3d7","Type":"ContainerDied","Data":"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"}
Oct 30 19:20:34.641877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.641828    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn" event={"ID":"8969b3a8-67af-408b-a1a5-d3657496c3d7","Type":"ContainerDied","Data":"e12feca8ecfcd059636c8ad3de38b190febdfc8c19f600c9202183cf37870541"}
Oct 30 19:20:34.641877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.641855    5065 scope.go:122] "RemoveContainer" containerID="fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"
Oct 30 19:20:34.643593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.643576    5065 generic.go:356] "Generic (PLEG): container finished" podID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" containerID="c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092" exitCode=0
Oct 30 19:20:34.643685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.643622    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"
Oct 30 19:20:34.643685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.643634    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" event={"ID":"f41130fe-bf8b-4ac8-8fd2-75bd98b68832","Type":"ContainerDied","Data":"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"}
Oct 30 19:20:34.643685 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.643676    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc" event={"ID":"f41130fe-bf8b-4ac8-8fd2-75bd98b68832","Type":"ContainerDied","Data":"6a80257a932a693e24a3823d89f5a4487c309deb0661b645cf34e42993ba2d9a"}
Oct 30 19:20:34.674172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.674154    5065 scope.go:122] "RemoveContainer" containerID="fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"
Oct 30 19:20:34.674472 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:34.674454    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc\": container with ID starting with fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc not found: ID does not exist" containerID="fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"
Oct 30 19:20:34.674553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.674478    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc"} err="failed to get container status \"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc\": rpc error: code = NotFound desc = could not find container \"fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc\": container with ID starting with fe1658e84ff085f1f9d84fcfdc3142f9831d074af7b9f1148fc318fa2b36e6cc not found: ID does not exist"
Oct 30 19:20:34.674553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.674494    5065 scope.go:122] "RemoveContainer" containerID="c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"
Oct 30 19:20:34.689247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.689223    5065 scope.go:122] "RemoveContainer" containerID="c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"
Oct 30 19:20:34.689596 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:34.689573    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092\": container with ID starting with c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092 not found: ID does not exist" containerID="c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"
Oct 30 19:20:34.689754 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.689606    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092"} err="failed to get container status \"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092\": rpc error: code = NotFound desc = could not find container \"c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092\": container with ID starting with c95ed245d07ede8b2223608ee8a7d8b1d5574e6ed61c57d6ed982f56badd4092 not found: ID does not exist"
Oct 30 19:20:34.691390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.691358    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.691540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.691522    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfh8l\" (UniqueName: \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.691641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.691562    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.691729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.691658    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlv8w\" (UniqueName: \"kubernetes.io/projected/8969b3a8-67af-408b-a1a5-d3657496c3d7-kube-api-access-nlv8w\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:34.692238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.692212    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.694268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.694233    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.697735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.697713    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:20:34.707928 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:20:34.707903    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e8531_f3a1_44c5_ac57_c2fa4a739bbe.slice/crio-f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445 WatchSource:0}: Error finding container f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445: Status 404 returned error can't find the container with id f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445
Oct 30 19:20:34.709525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.709500    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfh8l\" (UniqueName: \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\") pod \"maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:34.710565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.710547    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-de4bda-799c59d448-nqdsc"]
Oct 30 19:20:34.718854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.718832    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"]
Oct 30 19:20:34.727842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.727814    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:20:34.735253 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.735233    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-de4bda-75cc976787-s66nn"]
Oct 30 19:20:34.751982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:34.751931    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:35.148751 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:20:35.148727    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadd4fd4_0b63_4fbe_821e_8cd148440b11.slice/crio-dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9 WatchSource:0}: Error finding container dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9: Status 404 returned error can't find the container with id dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9
Oct 30 19:20:35.149447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.149423    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"]
Oct 30 19:20:35.654452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.654425    5065 generic.go:356] "Generic (PLEG): container finished" podID="636e8531-f3a1-44c5-ac57-c2fa4a739bbe" containerID="41be79748a77a7cf31e68746764934f7cd5c4841c93080baa19d292479b2673e" exitCode=0
Oct 30 19:20:35.654585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.654505    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9" event={"ID":"636e8531-f3a1-44c5-ac57-c2fa4a739bbe","Type":"ContainerDied","Data":"41be79748a77a7cf31e68746764934f7cd5c4841c93080baa19d292479b2673e"}
Oct 30 19:20:35.654585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.654569    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9" event={"ID":"636e8531-f3a1-44c5-ac57-c2fa4a739bbe","Type":"ContainerStarted","Data":"f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445"}
Oct 30 19:20:35.657359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.657333    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5" event={"ID":"badd4fd4-0b63-4fbe-821e-8cd148440b11","Type":"ContainerStarted","Data":"b3dc6081f4e01e683f892d8e40778df926c65a3c58643969681a123a3369f8dc"}
Oct 30 19:20:35.657454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.657383    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5" event={"ID":"badd4fd4-0b63-4fbe-821e-8cd148440b11","Type":"ContainerStarted","Data":"dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9"}
Oct 30 19:20:35.698879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.698832    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5" podStartSLOduration=1.69881862 podStartE2EDuration="1.69881862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:35.695872938 +0000 UTC m=+2855.959337888" watchObservedRunningTime="2026-10-30 19:20:35.69881862 +0000 UTC m=+2855.962283569"
Oct 30 19:20:35.899125 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.898811    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" path="/var/lib/kubelet/pods/8969b3a8-67af-408b-a1a5-d3657496c3d7/volumes"
Oct 30 19:20:35.901103 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:35.899633    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41130fe-bf8b-4ac8-8fd2-75bd98b68832" path="/var/lib/kubelet/pods/f41130fe-bf8b-4ac8-8fd2-75bd98b68832/volumes"
Oct 30 19:20:36.717109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:36.716491    5065 generic.go:356] "Generic (PLEG): container finished" podID="badd4fd4-0b63-4fbe-821e-8cd148440b11" containerID="b3dc6081f4e01e683f892d8e40778df926c65a3c58643969681a123a3369f8dc" exitCode=0
Oct 30 19:20:36.717313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:36.717136    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5" event={"ID":"badd4fd4-0b63-4fbe-821e-8cd148440b11","Type":"ContainerDied","Data":"b3dc6081f4e01e683f892d8e40778df926c65a3c58643969681a123a3369f8dc"}
Oct 30 19:20:37.086545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.086516    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:37.110478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.110445    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\") pod \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") "
Oct 30 19:20:37.110662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.110642    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\") pod \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") "
Oct 30 19:20:37.110846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.110826    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\" (UniqueName: \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\") pod \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\" (UID: \"636e8531-f3a1-44c5-ac57-c2fa4a739bbe\") "
Oct 30 19:20:37.111116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.111043    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca" pod "636e8531-f3a1-44c5-ac57-c2fa4a739bbe" (UID: "636e8531-f3a1-44c5-ac57-c2fa4a739bbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:37.111928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.111904    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.114228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.114200    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca" pod "636e8531-f3a1-44c5-ac57-c2fa4a739bbe" (UID: "636e8531-f3a1-44c5-ac57-c2fa4a739bbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:37.117318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.117292    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz" pod "636e8531-f3a1-44c5-ac57-c2fa4a739bbe" (UID: "636e8531-f3a1-44c5-ac57-c2fa4a739bbe"). InnerVolumeSpecName "kube-api-access-b5gvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:37.228208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.217679    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.228208 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.217721    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5gvz\" (UniqueName: \"kubernetes.io/projected/636e8531-f3a1-44c5-ac57-c2fa4a739bbe-kube-api-access-b5gvz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.728722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.728692    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"
Oct 30 19:20:37.728879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.728725    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9" event={"ID":"636e8531-f3a1-44c5-ac57-c2fa4a739bbe","Type":"ContainerDied","Data":"f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445"}
Oct 30 19:20:37.728879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.728773    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e26d313cda6c3a86094241853976be344b23451e0e4b40e21d5191da38b445"
Oct 30 19:20:37.937787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:37.937756    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:38.055110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.039103    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\" (UniqueName: \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\") pod \"badd4fd4-0b63-4fbe-821e-8cd148440b11\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") "
Oct 30 19:20:38.055110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.039334    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\") pod \"badd4fd4-0b63-4fbe-821e-8cd148440b11\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") "
Oct 30 19:20:38.055110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.039401    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\") pod \"badd4fd4-0b63-4fbe-821e-8cd148440b11\" (UID: \"badd4fd4-0b63-4fbe-821e-8cd148440b11\") "
Oct 30 19:20:38.055110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.049516    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca" pod "badd4fd4-0b63-4fbe-821e-8cd148440b11" (UID: "badd4fd4-0b63-4fbe-821e-8cd148440b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:38.055110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.053453    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l" pod "badd4fd4-0b63-4fbe-821e-8cd148440b11" (UID: "badd4fd4-0b63-4fbe-821e-8cd148440b11"). InnerVolumeSpecName "kube-api-access-hfh8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:38.068217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.064401    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca" pod "badd4fd4-0b63-4fbe-821e-8cd148440b11" (UID: "badd4fd4-0b63-4fbe-821e-8cd148440b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:38.141513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.141470    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.141513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.141509    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfh8l\" (UniqueName: \"kubernetes.io/projected/badd4fd4-0b63-4fbe-821e-8cd148440b11-kube-api-access-hfh8l\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.141513 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.141521    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badd4fd4-0b63-4fbe-821e-8cd148440b11-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.722135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.719980    5065 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:38.722135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.720040    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:38.739223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.739175    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5" event={"ID":"badd4fd4-0b63-4fbe-821e-8cd148440b11","Type":"ContainerDied","Data":"dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9"}
Oct 30 19:20:38.739223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.739206    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"
Oct 30 19:20:38.739435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:38.739245    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde9f5d19faa435940453b3ee07b791de56612272091b27d56b21f61005f89c9"
Oct 30 19:20:39.163621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.163595    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:20:39.258481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.258453    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.258711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.258574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.258860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.258758    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.258860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.258836    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.259055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.258978    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.259055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259036    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259054    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259153    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259245    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.259371    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\") pod \"f99cf479-590c-47e7-a6de-e1be93a0a0df\" (UID: \"f99cf479-590c-47e7-a6de-e1be93a0a0df\") "
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.260295    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99cf479-590c-47e7-a6de-e1be93a0a0df-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.260320    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.260401    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.260462    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.260758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.260671    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.261871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.261852    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.262176 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.262149    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "kube-api-access-zt7rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.262878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.262850    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:39.265501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.265474    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy" pod "f99cf479-590c-47e7-a6de-e1be93a0a0df" (UID: "f99cf479-590c-47e7-a6de-e1be93a0a0df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.362121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362063    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7rt\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-kube-api-access-zt7rt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362112    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362128    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362142    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362157    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362170    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.362330 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.362182    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99cf479-590c-47e7-a6de-e1be93a0a0df-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.799114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.798226    5065 generic.go:356] "Generic (PLEG): container finished" podID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerID="83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de" exitCode=0
Oct 30 19:20:39.799114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.798587    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" event={"ID":"f99cf479-590c-47e7-a6de-e1be93a0a0df","Type":"ContainerDied","Data":"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"}
Oct 30 19:20:39.799114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.798640    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq" event={"ID":"f99cf479-590c-47e7-a6de-e1be93a0a0df","Type":"ContainerDied","Data":"9449062b26b7b2f958427d3c728e85f8b21ad755f3c9669e451ff98c053df534"}
Oct 30 19:20:39.799114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.798670    5065 scope.go:122] "RemoveContainer" containerID="83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"
Oct 30 19:20:39.799114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.798821    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"
Oct 30 19:20:39.840735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.840709    5065 scope.go:122] "RemoveContainer" containerID="83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"
Oct 30 19:20:39.841052 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:39.841000    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de\": container with ID starting with 83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de not found: ID does not exist" containerID="83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"
Oct 30 19:20:39.841052 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.841040    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de"} err="failed to get container status \"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de\": rpc error: code = NotFound desc = could not find container \"83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de\": container with ID starting with 83c119f378e917e120bef7326e1c0e1d273a4b6ea0f8faa2c1f0fc77bb6262de not found: ID does not exist"
Oct 30 19:20:39.848318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.848285    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:20:39.853889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.853867    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-wg4kq"]
Oct 30 19:20:39.901875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:39.898745    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" path="/var/lib/kubelet/pods/f99cf479-590c-47e7-a6de-e1be93a0a0df/volumes"
Oct 30 19:20:43.841380 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:43.841324    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 19:20:44.048654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:44.048616    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"]
Oct 30 19:20:44.056843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:44.056804    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-86c150-sv26l"]
Oct 30 19:20:45.904395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:45.904356    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2641bd26-6a09-474d-90d3-e8a6403a651c" path="/var/lib/kubelet/pods/2641bd26-6a09-474d-90d3-e8a6403a651c/volumes"
Oct 30 19:20:48.765130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:48.764437    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"]
Oct 30 19:20:48.772967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:48.772935    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7c6027-z7jq5"]
Oct 30 19:20:48.860112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:48.850167    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing" containerID="cri-o://7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e" gracePeriod=25
Oct 30 19:20:49.894158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:49.894121    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badd4fd4-0b63-4fbe-821e-8cd148440b11" path="/var/lib/kubelet/pods/badd4fd4-0b63-4fbe-821e-8cd148440b11/volumes"
Oct 30 19:20:51.261417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:51.261377    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:52.921977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:52.908736    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:20:53.187175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.179944    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" containerID="cri-o://2ed9a1ee27eb30c83ed26ded5c51ae29b610a40061c5b037770189f20b7d9fdb" gracePeriod=30
Oct 30 19:20:53.272318 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.271685    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"]
Oct 30 19:20:53.440674 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.440545    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-784fbf-8grc9"]
Oct 30 19:20:53.614461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.614418    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"]
Oct 30 19:20:53.615594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615565    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy"
Oct 30 19:20:53.615594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615594    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615623    5065 cpu_manager.go:414] "containerMap: removing container" podUID="636e8531-f3a1-44c5-ac57-c2fa4a739bbe" containerName="revoke-keys"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615631    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636e8531-f3a1-44c5-ac57-c2fa4a739bbe" containerName="revoke-keys"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615662    5065 cpu_manager.go:414] "containerMap: removing container" podUID="badd4fd4-0b63-4fbe-821e-8cd148440b11" containerName="revoke-keys"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615670    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badd4fd4-0b63-4fbe-821e-8cd148440b11" containerName="revoke-keys"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615722    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing"
Oct 30 19:20:53.615805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615730    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing"
Oct 30 19:20:53.616151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615902    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99cf479-590c-47e7-a6de-e1be93a0a0df" containerName="istio-proxy"
Oct 30 19:20:53.616151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615919    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="636e8531-f3a1-44c5-ac57-c2fa4a739bbe" containerName="revoke-keys"
Oct 30 19:20:53.616151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615932    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="badd4fd4-0b63-4fbe-821e-8cd148440b11" containerName="revoke-keys"
Oct 30 19:20:53.616151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.615950    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8969b3a8-67af-408b-a1a5-d3657496c3d7" containerName="payload-pre-processing"
Oct 30 19:20:53.638405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.638381    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"]
Oct 30 19:20:53.638672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.638658    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.661434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.661390    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:20:53.661794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.661777    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:53.673823 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:20:53.673790    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:20:53.688209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.687797    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.688209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.687909    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.688209 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.688034    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4fc\" (UniqueName: \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.789871 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.789832    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4fc\" (UniqueName: \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.790155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.790131    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.790275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.790240    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.791031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.791004    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.793625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.793599    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.806681 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.806657    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4fc\" (UniqueName: \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:53.927385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.925906    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636e8531-f3a1-44c5-ac57-c2fa4a739bbe" path="/var/lib/kubelet/pods/636e8531-f3a1-44c5-ac57-c2fa4a739bbe/volumes"
Oct 30 19:20:53.959166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:53.959141    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:54.276044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:54.275999    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"]
Oct 30 19:20:54.609313 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:54.609276    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp" event={"ID":"22af931a-38fb-48b4-b52d-62bba5226c4f","Type":"ContainerStarted","Data":"500057bb2f740adc09a0fc4e5b7dd196672339d9cd7e61a5ad4e6585687657dd"}
Oct 30 19:20:55.622142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:55.622033    5065 generic.go:356] "Generic (PLEG): container finished" podID="22af931a-38fb-48b4-b52d-62bba5226c4f" containerID="5b8c96a83f20d0c09835a87c3e8733652d7b006d5191bdb92e3e44c312ce1df9" exitCode=0
Oct 30 19:20:55.622142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:55.622077    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp" event={"ID":"22af931a-38fb-48b4-b52d-62bba5226c4f","Type":"ContainerDied","Data":"5b8c96a83f20d0c09835a87c3e8733652d7b006d5191bdb92e3e44c312ce1df9"}
Oct 30 19:20:55.716119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:55.716042    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:55.716119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:55.716116    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:56.808299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.808277    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:56.951433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.951355    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\") pod \"22af931a-38fb-48b4-b52d-62bba5226c4f\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") "
Oct 30 19:20:56.951625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.951497    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\") pod \"22af931a-38fb-48b4-b52d-62bba5226c4f\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") "
Oct 30 19:20:56.951625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.951518    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\" (UniqueName: \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\") pod \"22af931a-38fb-48b4-b52d-62bba5226c4f\" (UID: \"22af931a-38fb-48b4-b52d-62bba5226c4f\") "
Oct 30 19:20:56.951864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.951836    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca" pod "22af931a-38fb-48b4-b52d-62bba5226c4f" (UID: "22af931a-38fb-48b4-b52d-62bba5226c4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:56.952026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.952012    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:56.953722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.953698    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca" pod "22af931a-38fb-48b4-b52d-62bba5226c4f" (UID: "22af931a-38fb-48b4-b52d-62bba5226c4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:56.953820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:56.953730    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc" pod "22af931a-38fb-48b4-b52d-62bba5226c4f" (UID: "22af931a-38fb-48b4-b52d-62bba5226c4f"). InnerVolumeSpecName "kube-api-access-vd4fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:57.053738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.053704    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22af931a-38fb-48b4-b52d-62bba5226c4f-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:57.053738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.053736    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4fc\" (UniqueName: \"kubernetes.io/projected/22af931a-38fb-48b4-b52d-62bba5226c4f-kube-api-access-vd4fc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:57.303885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.303849    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:20:57.304143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.304114    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy" containerID="cri-o://0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37" gracePeriod=30
Oct 30 19:20:57.659630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.659547    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp" event={"ID":"22af931a-38fb-48b4-b52d-62bba5226c4f","Type":"ContainerDied","Data":"500057bb2f740adc09a0fc4e5b7dd196672339d9cd7e61a5ad4e6585687657dd"}
Oct 30 19:20:57.659630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.659596    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500057bb2f740adc09a0fc4e5b7dd196672339d9cd7e61a5ad4e6585687657dd"
Oct 30 19:20:57.659840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.659645    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"
Oct 30 19:20:57.858241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.858204    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"]
Oct 30 19:20:57.859321 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.859303    5065 cpu_manager.go:414] "containerMap: removing container" podUID="22af931a-38fb-48b4-b52d-62bba5226c4f" containerName="revoke-keys"
Oct 30 19:20:57.859378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.859323    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22af931a-38fb-48b4-b52d-62bba5226c4f" containerName="revoke-keys"
Oct 30 19:20:57.859557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.859546    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="22af931a-38fb-48b4-b52d-62bba5226c4f" containerName="revoke-keys"
Oct 30 19:20:57.863863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.863841    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:57.879189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.879164    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:57.879324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.879296    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:20:57.902976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.902942    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"]
Oct 30 19:20:57.997067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.996981    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:57.997067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.997024    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:57.997543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:57.997511    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhww\" (UniqueName: \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.099143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.099101    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhww\" (UniqueName: \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.099328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.099236    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.099328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.099274    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.100013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.099988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.101939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.101911    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.112647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.112610    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhww\" (UniqueName: \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.179123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.179099    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:20:58.513596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.513524    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"]
Oct 30 19:20:58.672774 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.672741    5065 generic.go:356] "Generic (PLEG): container finished" podID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerID="2ed9a1ee27eb30c83ed26ded5c51ae29b610a40061c5b037770189f20b7d9fdb" exitCode=0
Oct 30 19:20:58.672955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.672815    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" event={"ID":"51f029b4-0921-4f4a-bc47-25d2920bf00f","Type":"ContainerDied","Data":"2ed9a1ee27eb30c83ed26ded5c51ae29b610a40061c5b037770189f20b7d9fdb"}
Oct 30 19:20:58.674750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.674728    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:20:58.674917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.674782    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4" event={"ID":"cd45e1f2-bd36-4093-b636-96e22d1b9fea","Type":"ContainerStarted","Data":"6d677c6cdb1b34f2298f20550fad522a58688ecb8be56d6216fbf885477a92b6"}
Oct 30 19:20:58.783668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783635    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.783852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783696    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.783852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783750    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.783852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783805    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.783968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783953    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.784011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.783989    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.784113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.784067    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.784152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.784117    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.784292 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.784278    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\") pod \"51f029b4-0921-4f4a-bc47-25d2920bf00f\" (UID: \"51f029b4-0921-4f4a-bc47-25d2920bf00f\") "
Oct 30 19:20:58.785353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.785324    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.786437 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.786407    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.786620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.786597    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.786703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.786689    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.787223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.787194    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:58.787310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.787198    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:58.787531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.787497    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.788401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.788378    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:58.789018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.788991    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm" pod "51f029b4-0921-4f4a-bc47-25d2920bf00f" (UID: "51f029b4-0921-4f4a-bc47-25d2920bf00f"). InnerVolumeSpecName "kube-api-access-snqkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:58.887433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887400    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887436    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887452    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887466    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887480    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887492    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f029b4-0921-4f4a-bc47-25d2920bf00f-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887505    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqkm\" (UniqueName: \"kubernetes.io/projected/51f029b4-0921-4f4a-bc47-25d2920bf00f-kube-api-access-snqkm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887518    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f029b4-0921-4f4a-bc47-25d2920bf00f-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.887878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:58.887530    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f029b4-0921-4f4a-bc47-25d2920bf00f-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:59.685233 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.685193    5065 generic.go:356] "Generic (PLEG): container finished" podID="cd45e1f2-bd36-4093-b636-96e22d1b9fea" containerID="c1af2e43bed7eb21040342b47e4be1884dd73ff04f5f9719daa39023a241b1cf" exitCode=0
Oct 30 19:20:59.685416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.685280    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4" event={"ID":"cd45e1f2-bd36-4093-b636-96e22d1b9fea","Type":"ContainerDied","Data":"c1af2e43bed7eb21040342b47e4be1884dd73ff04f5f9719daa39023a241b1cf"}
Oct 30 19:20:59.687161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.687139    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"
Oct 30 19:20:59.687161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.687144    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx" event={"ID":"51f029b4-0921-4f4a-bc47-25d2920bf00f","Type":"ContainerDied","Data":"bfca57e4144c5bdcdadb6ddd573a9f287c4210b267580396e45821a70d6edced"}
Oct 30 19:20:59.687376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.687203    5065 scope.go:122] "RemoveContainer" containerID="2ed9a1ee27eb30c83ed26ded5c51ae29b610a40061c5b037770189f20b7d9fdb"
Oct 30 19:20:59.746855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.746823    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:20:59.754062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.754035    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-lvbdx"]
Oct 30 19:20:59.896499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:20:59.896412    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" path="/var/lib/kubelet/pods/51f029b4-0921-4f4a-bc47-25d2920bf00f/volumes"
Oct 30 19:21:00.896902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:00.896868    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:21:01.030154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.030120    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\") pod \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") "
Oct 30 19:21:01.030363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.030215    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\" (UniqueName: \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\") pod \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") "
Oct 30 19:21:01.030446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.030371    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\") pod \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\" (UID: \"cd45e1f2-bd36-4093-b636-96e22d1b9fea\") "
Oct 30 19:21:01.030777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.030750    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca" pod "cd45e1f2-bd36-4093-b636-96e22d1b9fea" (UID: "cd45e1f2-bd36-4093-b636-96e22d1b9fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:01.031320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.031297    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.032416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.032389    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca" pod "cd45e1f2-bd36-4093-b636-96e22d1b9fea" (UID: "cd45e1f2-bd36-4093-b636-96e22d1b9fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:01.032612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.032589    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww" pod "cd45e1f2-bd36-4093-b636-96e22d1b9fea" (UID: "cd45e1f2-bd36-4093-b636-96e22d1b9fea"). InnerVolumeSpecName "kube-api-access-5hhww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:01.132560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.132532    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd45e1f2-bd36-4093-b636-96e22d1b9fea-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.132560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.132557    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhww\" (UniqueName: \"kubernetes.io/projected/cd45e1f2-bd36-4093-b636-96e22d1b9fea-kube-api-access-5hhww\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.319683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.318651    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:01.708026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.707868    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4" event={"ID":"cd45e1f2-bd36-4093-b636-96e22d1b9fea","Type":"ContainerDied","Data":"6d677c6cdb1b34f2298f20550fad522a58688ecb8be56d6216fbf885477a92b6"}
Oct 30 19:21:01.708319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.708293    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d677c6cdb1b34f2298f20550fad522a58688ecb8be56d6216fbf885477a92b6"
Oct 30 19:21:01.709040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:01.707832    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"
Oct 30 19:21:02.073531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.073495    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:21:02.074011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.073754    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy" containerID="cri-o://f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca" gracePeriod=30
Oct 30 19:21:02.690426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.690400    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:21:02.751275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.743021    5065 generic.go:356] "Generic (PLEG): container finished" podID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerID="0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37" exitCode=0
Oct 30 19:21:02.751275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.743237    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerDied","Data":"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"}
Oct 30 19:21:02.751275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.743351    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb" event={"ID":"dce02056-7b6e-4a5c-8cc2-3982eebb3e08","Type":"ContainerDied","Data":"aadc262a8fd1f5cb624f0249841abd1b8f7a521611568c054d00bac5244dde20"}
Oct 30 19:21:02.751275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.743392    5065 scope.go:122] "RemoveContainer" containerID="0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"
Oct 30 19:21:02.751275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.743619    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"
Oct 30 19:21:02.792501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.792459    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"]
Oct 30 19:21:02.793657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793630    5065 cpu_manager.go:414] "containerMap: removing container" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:02.793740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793660    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:02.793740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793689    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cd45e1f2-bd36-4093-b636-96e22d1b9fea" containerName="revoke-keys"
Oct 30 19:21:02.793740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793697    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd45e1f2-bd36-4093-b636-96e22d1b9fea" containerName="revoke-keys"
Oct 30 19:21:02.793909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793745    5065 cpu_manager.go:414] "containerMap: removing container" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy"
Oct 30 19:21:02.793909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793753    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy"
Oct 30 19:21:02.794012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793946    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:02.794012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793963    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd45e1f2-bd36-4093-b636-96e22d1b9fea" containerName="revoke-keys"
Oct 30 19:21:02.794012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.793979    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f029b4-0921-4f4a-bc47-25d2920bf00f" containerName="istio-proxy"
Oct 30 19:21:02.794485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.794463    5065 scope.go:122] "RemoveContainer" containerID="2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"
Oct 30 19:21:02.795522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795497    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795561    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795581    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795722    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795774    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795797    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795846    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795901    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.795954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.795937    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\") pod \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\" (UID: \"dce02056-7b6e-4a5c-8cc2-3982eebb3e08\") "
Oct 30 19:21:02.796594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.796573    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.796851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.796736    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.797970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.797614    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:02.797970 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.797857    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.807670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.798048    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.812408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.810486    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"]
Oct 30 19:21:02.812408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.810619    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:02.816749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.815293    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.816749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.815355    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "kube-api-access-78k9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.816749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.815362    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:02.816749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.815570    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token" pod "dce02056-7b6e-4a5c-8cc2-3982eebb3e08" (UID: "dce02056-7b6e-4a5c-8cc2-3982eebb3e08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.839115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.838348    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:02.839115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.838451    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:02.855354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.855318    5065 scope.go:122] "RemoveContainer" containerID="0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"
Oct 30 19:21:02.855814 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:02.855784    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37\": container with ID starting with 0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37 not found: ID does not exist" containerID="0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"
Oct 30 19:21:02.855922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.855827    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37"} err="failed to get container status \"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37\": rpc error: code = NotFound desc = could not find container \"0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37\": container with ID starting with 0472e881c470d8095ebccf1e3c7dbf7c47f950f9ed4a26b3359ae1c510895d37 not found: ID does not exist"
Oct 30 19:21:02.855922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.855854    5065 scope.go:122] "RemoveContainer" containerID="2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"
Oct 30 19:21:02.856356 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:02.856254    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b\": container with ID starting with 2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b not found: ID does not exist" containerID="2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"
Oct 30 19:21:02.856356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.856293    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b"} err="failed to get container status \"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b\": rpc error: code = NotFound desc = could not find container \"2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b\": container with ID starting with 2376b878f6a5d48d392b30cc8af29aa84ff50485c92413fa5309a7802981cc0b not found: ID does not exist"
Oct 30 19:21:02.897668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.897627    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfz4d\" (UniqueName: \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:02.897892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.897868    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:02.898194 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898161    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:02.898280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898259    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898276    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898292    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78k9z\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-kube-api-access-78k9z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898309    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898324    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898338    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898352    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898366    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.898390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:02.898386    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce02056-7b6e-4a5c-8cc2-3982eebb3e08-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:03.012409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.008224    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.012409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.008330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfz4d\" (UniqueName: \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.012409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.008477    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.012409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.009484    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.024108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.016476    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.043108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.038273    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfz4d\" (UniqueName: \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\") pod \"maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.166221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.166164    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:03.173801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.173773    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:21:03.183217 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.181405    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-t2fcb"]
Oct 30 19:21:03.516412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:03.516340    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"]
Oct 30 19:21:04.010356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:04.010313    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" path="/var/lib/kubelet/pods/dce02056-7b6e-4a5c-8cc2-3982eebb3e08/volumes"
Oct 30 19:21:04.011413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:04.011388    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb" event={"ID":"65908435-5699-4ea0-ac57-e39865463239","Type":"ContainerStarted","Data":"5cf3e1f4027b3d9e01bba463dfefcb56c78b6b86455b5a89e3ec569c5f3be463"}
Oct 30 19:21:05.025800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:05.025766    5065 generic.go:356] "Generic (PLEG): container finished" podID="65908435-5699-4ea0-ac57-e39865463239" containerID="4aebb7189782b4a2513d31755660ac4fb11072d596bace95ec03a7d1b8762d57" exitCode=0
Oct 30 19:21:05.026235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:05.025850    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb" event={"ID":"65908435-5699-4ea0-ac57-e39865463239","Type":"ContainerDied","Data":"4aebb7189782b4a2513d31755660ac4fb11072d596bace95ec03a7d1b8762d57"}
Oct 30 19:21:06.295425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.295362    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:06.479031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.478999    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\") pod \"65908435-5699-4ea0-ac57-e39865463239\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") "
Oct 30 19:21:06.479274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.479134    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\") pod \"65908435-5699-4ea0-ac57-e39865463239\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") "
Oct 30 19:21:06.479274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.479236    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\" (UniqueName: \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\") pod \"65908435-5699-4ea0-ac57-e39865463239\" (UID: \"65908435-5699-4ea0-ac57-e39865463239\") "
Oct 30 19:21:06.479474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.479445    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca" pod "65908435-5699-4ea0-ac57-e39865463239" (UID: "65908435-5699-4ea0-ac57-e39865463239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:06.479816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.479797    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65908435-5699-4ea0-ac57-e39865463239-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:06.482043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.482016    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca" pod "65908435-5699-4ea0-ac57-e39865463239" (UID: "65908435-5699-4ea0-ac57-e39865463239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:06.482197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.482050    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d" pod "65908435-5699-4ea0-ac57-e39865463239" (UID: "65908435-5699-4ea0-ac57-e39865463239"). InnerVolumeSpecName "kube-api-access-tfz4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:06.580819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.580729    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65908435-5699-4ea0-ac57-e39865463239-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:06.580819 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:06.580769    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfz4d\" (UniqueName: \"kubernetes.io/projected/65908435-5699-4ea0-ac57-e39865463239-kube-api-access-tfz4d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.308737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.308705    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb" event={"ID":"65908435-5699-4ea0-ac57-e39865463239","Type":"ContainerDied","Data":"5cf3e1f4027b3d9e01bba463dfefcb56c78b6b86455b5a89e3ec569c5f3be463"}
Oct 30 19:21:07.320642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.320620    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cf3e1f4027b3d9e01bba463dfefcb56c78b6b86455b5a89e3ec569c5f3be463"
Oct 30 19:21:07.320833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.320554    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"
Oct 30 19:21:07.588421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.588394    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:21:07.637066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.637027    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.637574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638024    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638161    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638212    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638297    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638349    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638421    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.638656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.638477    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\") pod \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\" (UID: \"6337fbdf-bb56-480e-adea-4f20ca6b2a5f\") "
Oct 30 19:21:07.639597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.639488    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:07.640152 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.640020    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.640976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.640382    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.640976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.640700    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.640976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.640808    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.641367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.641312    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.641594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.641547    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:07.641940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.641899    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.644517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.644492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "kube-api-access-fs5l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:07.673415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.673374    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token" pod "6337fbdf-bb56-480e-adea-4f20ca6b2a5f" (UID: "6337fbdf-bb56-480e-adea-4f20ca6b2a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:07.741626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741597    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741628    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs5l7\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-kube-api-access-fs5l7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741639    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741647    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741657    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741667    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741682    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.741812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:07.741692    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6337fbdf-bb56-480e-adea-4f20ca6b2a5f-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:08.109014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.108983    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 19:21:08.120558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.120527    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 19:21:08.320719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.320624    5065 generic.go:356] "Generic (PLEG): container finished" podID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerID="f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca" exitCode=0
Oct 30 19:21:08.320719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.320671    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"
Oct 30 19:21:08.320719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.320704    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" event={"ID":"6337fbdf-bb56-480e-adea-4f20ca6b2a5f","Type":"ContainerDied","Data":"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"}
Oct 30 19:21:08.321384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.320766    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns" event={"ID":"6337fbdf-bb56-480e-adea-4f20ca6b2a5f","Type":"ContainerDied","Data":"22e9c6029f89b89bcbb0c88cdb788ef4a7718a0d3283e240e546bb298d0c94c9"}
Oct 30 19:21:08.321384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.320792    5065 scope.go:122] "RemoveContainer" containerID="f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"
Oct 30 19:21:08.334583 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.334552    5065 scope.go:122] "RemoveContainer" containerID="f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"
Oct 30 19:21:08.334860 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:08.334836    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca\": container with ID starting with f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca not found: ID does not exist" containerID="f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"
Oct 30 19:21:08.334947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.334867    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca"} err="failed to get container status \"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca\": rpc error: code = NotFound desc = could not find container \"f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca\": container with ID starting with f39a762e9d952f61be341461f50d53c303e338a8a44e449f9c3e64d3c7351aca not found: ID does not exist"
Oct 30 19:21:08.351406 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.351380    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:21:08.358860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:08.358841    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-67mns"]
Oct 30 19:21:09.178964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.178932    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 19:21:09.300424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.300390    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\") pod \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") "
Oct 30 19:21:09.300715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.300691    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume" pod "a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" (UID: "a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:09.300784 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.300702    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\" (UniqueName: \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\") pod \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\" (UID: \"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb\") "
Oct 30 19:21:09.301309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.301288    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:09.303056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.303026    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct" pod "a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" (UID: "a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb"). InnerVolumeSpecName "kube-api-access-tbxct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:09.334853 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.334825    5065 generic.go:356] "Generic (PLEG): container finished" podID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerID="7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e" exitCode=0
Oct 30 19:21:09.335237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.334888    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"
Oct 30 19:21:09.335237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.334909    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" event={"ID":"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb","Type":"ContainerDied","Data":"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"}
Oct 30 19:21:09.335237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.334961    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq" event={"ID":"a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb","Type":"ContainerDied","Data":"37319da8f3411a3f1df965138381c214c09321cb667dbd7e829a641d551c75fa"}
Oct 30 19:21:09.335237 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.334986    5065 scope.go:122] "RemoveContainer" containerID="7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"
Oct 30 19:21:09.347382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.347366    5065 scope.go:122] "RemoveContainer" containerID="7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"
Oct 30 19:21:09.347644 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:09.347624    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e\": container with ID starting with 7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e not found: ID does not exist" containerID="7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"
Oct 30 19:21:09.347700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.347651    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e"} err="failed to get container status \"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e\": rpc error: code = NotFound desc = could not find container \"7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e\": container with ID starting with 7530bc7809faf85e05cbcd540da6bd8c10c7af2b9b10187aa62e657a1fb7866e not found: ID does not exist"
Oct 30 19:21:09.373079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.373055    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 19:21:09.398204 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.398184    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-784fbf-575447c7bc-hj7tq"]
Oct 30 19:21:09.402985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.402934    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxct\" (UniqueName: \"kubernetes.io/projected/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb-kube-api-access-tbxct\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:09.919736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.919691    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" path="/var/lib/kubelet/pods/6337fbdf-bb56-480e-adea-4f20ca6b2a5f/volumes"
Oct 30 19:21:09.920676 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:09.920649    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" path="/var/lib/kubelet/pods/a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb/volumes"
Oct 30 19:21:12.841338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:12.841308    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 19:21:12.867841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:12.867808    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 19:21:13.109536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:13.109424    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing" containerID="cri-o://f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4" gracePeriod=25
Oct 30 19:21:13.121588 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:13.121535    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing" containerID="cri-o://a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d" gracePeriod=25
Oct 30 19:21:13.601048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:13.601005    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"]
Oct 30 19:21:13.610862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:13.610826    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-03362d-djmrp"]
Oct 30 19:21:13.893141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:13.893051    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22af931a-38fb-48b4-b52d-62bba5226c4f" path="/var/lib/kubelet/pods/22af931a-38fb-48b4-b52d-62bba5226c4f/volumes"
Oct 30 19:21:14.319627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:14.319470    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:14.319627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:14.319574    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:15.700111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.699149    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"]
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700307    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700332    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700369    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700377    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700405    5065 cpu_manager.go:414] "containerMap: removing container" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700412    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700462    5065 cpu_manager.go:414] "containerMap: removing container" podUID="65908435-5699-4ea0-ac57-e39865463239" containerName="revoke-keys"
Oct 30 19:21:15.700585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700469    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65908435-5699-4ea0-ac57-e39865463239" containerName="revoke-keys"
Oct 30 19:21:15.700964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700630    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b06930-7ae3-4289-a0f6-f90b8e9fe6bb" containerName="payload-pre-processing"
Oct 30 19:21:15.700964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700644    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6337fbdf-bb56-480e-adea-4f20ca6b2a5f" containerName="istio-proxy"
Oct 30 19:21:15.700964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700657    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce02056-7b6e-4a5c-8cc2-3982eebb3e08" containerName="istio-proxy"
Oct 30 19:21:15.700964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.700674    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="65908435-5699-4ea0-ac57-e39865463239" containerName="revoke-keys"
Oct 30 19:21:15.762336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.762297    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"]
Oct 30 19:21:15.762687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.762663    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:15.778467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.778437    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:15.799335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.799296    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:15.939952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.939919    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:15.940191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.940168    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l548d\" (UniqueName: \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:15.940683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:15.940662    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.095036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.095009    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.095161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.095051    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l548d\" (UniqueName: \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.095246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.095204    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.095616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.095590    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.098366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.098342    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.108994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.108962    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l548d\" (UniqueName: \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\") pod \"maas-api-revoke-sub-04d6e3d1-z7n6n\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.308841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.308806    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:16.608313 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:16.608274    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5692a780_8fdd_484f_a30c_d127de618e4a.slice/crio-d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d WatchSource:0}: Error finding container d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d: Status 404 returned error can't find the container with id d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d
Oct 30 19:21:16.609584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:16.609385    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"]
Oct 30 19:21:17.513627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:17.513593    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n" event={"ID":"5692a780-8fdd-484f-a30c-d127de618e4a","Type":"ContainerStarted","Data":"106ca2078bda4e8d236825052e95b476fa2d48b96824b13b9edc24b9159f2404"}
Oct 30 19:21:17.514035 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:17.513651    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n" event={"ID":"5692a780-8fdd-484f-a30c-d127de618e4a","Type":"ContainerStarted","Data":"d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d"}
Oct 30 19:21:17.542622 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:17.542572    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n" podStartSLOduration=2.542559286 podStartE2EDuration="2.542559286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:17.539235154 +0000 UTC m=+2897.802700097" watchObservedRunningTime="2026-10-30 19:21:17.542559286 +0000 UTC m=+2897.806024255"
Oct 30 19:21:17.842689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:17.842606    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing" containerID="cri-o://d82c7f9f774ed0f309e4b74b8f06e0a6f0a630203e2f556ab9cfe3b0c3e17f99" gracePeriod=25
Oct 30 19:21:17.868460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:17.868410    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing" containerID="cri-o://7b507b7880e76f7c400445d8683e20c9511e038865272893cf70fe9c3ffd1a80" gracePeriod=25
Oct 30 19:21:18.523226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:18.523192    5065 generic.go:356] "Generic (PLEG): container finished" podID="5692a780-8fdd-484f-a30c-d127de618e4a" containerID="106ca2078bda4e8d236825052e95b476fa2d48b96824b13b9edc24b9159f2404" exitCode=0
Oct 30 19:21:18.523644 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:18.523273    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n" event={"ID":"5692a780-8fdd-484f-a30c-d127de618e4a","Type":"ContainerDied","Data":"106ca2078bda4e8d236825052e95b476fa2d48b96824b13b9edc24b9159f2404"}
Oct 30 19:21:19.876166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.851340    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:19.985450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.985332    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\" (UniqueName: \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\") pod \"5692a780-8fdd-484f-a30c-d127de618e4a\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") "
Oct 30 19:21:19.985650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.985503    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\") pod \"5692a780-8fdd-484f-a30c-d127de618e4a\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") "
Oct 30 19:21:19.985730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.985687    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\") pod \"5692a780-8fdd-484f-a30c-d127de618e4a\" (UID: \"5692a780-8fdd-484f-a30c-d127de618e4a\") "
Oct 30 19:21:19.989636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.986428    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca" pod "5692a780-8fdd-484f-a30c-d127de618e4a" (UID: "5692a780-8fdd-484f-a30c-d127de618e4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:19.989636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.986817    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.999459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:19.997032    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"]
Oct 30 19:21:20.006157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.005150    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-304f22-wddk4"]
Oct 30 19:21:20.006157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.005464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca" pod "5692a780-8fdd-484f-a30c-d127de618e4a" (UID: "5692a780-8fdd-484f-a30c-d127de618e4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:20.006157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.005511    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d" pod "5692a780-8fdd-484f-a30c-d127de618e4a" (UID: "5692a780-8fdd-484f-a30c-d127de618e4a"). InnerVolumeSpecName "kube-api-access-l548d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:20.135161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.132483    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5692a780-8fdd-484f-a30c-d127de618e4a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.135161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.132524    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l548d\" (UniqueName: \"kubernetes.io/projected/5692a780-8fdd-484f-a30c-d127de618e4a-kube-api-access-l548d\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.547635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.547603    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"
Oct 30 19:21:20.547821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.547653    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n" event={"ID":"5692a780-8fdd-484f-a30c-d127de618e4a","Type":"ContainerDied","Data":"d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d"}
Oct 30 19:21:20.547821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:20.547711    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86a9f320823d6d7426f059db4c670e980d80607835110bef8662e1fc01d849d"
Oct 30 19:21:21.891377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:21.891346    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd45e1f2-bd36-4093-b636-96e22d1b9fea" path="/var/lib/kubelet/pods/cd45e1f2-bd36-4093-b636-96e22d1b9fea/volumes"
Oct 30 19:21:22.047040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:22.046999    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:22.047215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:22.047041    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:22.405114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:22.405042    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 19:21:22.405386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:22.405319    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy" containerID="cri-o://f82fb8963a5bc6bdcaf6863460b5b9625c032803fa693c9154bf3efcac849ef6" gracePeriod=30
Oct 30 19:21:23.070991 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:23.070952    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:21:23.221153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.221119    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"]
Oct 30 19:21:23.222360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.222338    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5692a780-8fdd-484f-a30c-d127de618e4a" containerName="revoke-keys"
Oct 30 19:21:23.222427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.222364    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5692a780-8fdd-484f-a30c-d127de618e4a" containerName="revoke-keys"
Oct 30 19:21:23.222545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.222532    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5692a780-8fdd-484f-a30c-d127de618e4a" containerName="revoke-keys"
Oct 30 19:21:23.231734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.228549    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.233142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.233116    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"]
Oct 30 19:21:23.235472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.235450    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:23.235595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.235503    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:23.332403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.332314    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.332608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.332585    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfwl\" (UniqueName: \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.332690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.332646    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.433696 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.433651    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfwl\" (UniqueName: \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.433928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.433722    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.433928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.433869    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.434592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.434561    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.436279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.436254    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.447621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.447597    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfwl\" (UniqueName: \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\") pod \"maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.547791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.547754    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:23.857438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:23.857407    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"]
Oct 30 19:21:23.858358 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:23.858326    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode800cee6_7d63_49e8_95de_a4825e15e142.slice/crio-2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63 WatchSource:0}: Error finding container 2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63: Status 404 returned error can't find the container with id 2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63
Oct 30 19:21:24.320438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:24.320315    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:24.320438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:24.320425    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:24.661688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:24.661649    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n" event={"ID":"e800cee6-7d63-49e8-95de-a4825e15e142","Type":"ContainerStarted","Data":"59786920d986b9b0c70264b677b0bfe76441144990f9b106215e600591de6e7b"}
Oct 30 19:21:24.661873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:24.661710    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n" event={"ID":"e800cee6-7d63-49e8-95de-a4825e15e142","Type":"ContainerStarted","Data":"2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63"}
Oct 30 19:21:24.688845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:24.688798    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n" podStartSLOduration=1.688784771 podStartE2EDuration="1.688784771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:24.685828659 +0000 UTC m=+2904.949293607" watchObservedRunningTime="2026-10-30 19:21:24.688784771 +0000 UTC m=+2904.952249708"
Oct 30 19:21:25.673156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:25.673076    5065 generic.go:356] "Generic (PLEG): container finished" podID="e800cee6-7d63-49e8-95de-a4825e15e142" containerID="59786920d986b9b0c70264b677b0bfe76441144990f9b106215e600591de6e7b" exitCode=0
Oct 30 19:21:25.673573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:25.673181    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n" event={"ID":"e800cee6-7d63-49e8-95de-a4825e15e142","Type":"ContainerDied","Data":"59786920d986b9b0c70264b677b0bfe76441144990f9b106215e600591de6e7b"}
Oct 30 19:21:26.228448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.228410    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"]
Oct 30 19:21:26.233189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.233168    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.250712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.250681    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"]
Oct 30 19:21:26.279503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.279473    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6w75\" (UniqueName: \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.279646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.279528    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.279902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.279880    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.381163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.381134    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.381370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.381219    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6w75\" (UniqueName: \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.381370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.381240    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.381969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.381889    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.383949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.383926    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.395728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.395705    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6w75\" (UniqueName: \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\") pod \"maas-api-revoke-sub-54b345b4-qrq69\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.550142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.550057    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:26.848980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.848951    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"]
Oct 30 19:21:26.849348 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:26.849206    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebd859c_e39c_4350_a294_7f99ddcedc61.slice/crio-6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5 WatchSource:0}: Error finding container 6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5: Status 404 returned error can't find the container with id 6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5
Oct 30 19:21:26.866459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.866436    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:26.889528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.889501    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\" (UniqueName: \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\") pod \"e800cee6-7d63-49e8-95de-a4825e15e142\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") "
Oct 30 19:21:26.889651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.889598    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\") pod \"e800cee6-7d63-49e8-95de-a4825e15e142\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") "
Oct 30 19:21:26.889727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.889655    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\") pod \"e800cee6-7d63-49e8-95de-a4825e15e142\" (UID: \"e800cee6-7d63-49e8-95de-a4825e15e142\") "
Oct 30 19:21:26.890042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.890013    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca" pod "e800cee6-7d63-49e8-95de-a4825e15e142" (UID: "e800cee6-7d63-49e8-95de-a4825e15e142"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:26.890490 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.890469    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:26.892356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.892319    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl" pod "e800cee6-7d63-49e8-95de-a4825e15e142" (UID: "e800cee6-7d63-49e8-95de-a4825e15e142"). InnerVolumeSpecName "kube-api-access-tsfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:26.893043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.893021    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca" pod "e800cee6-7d63-49e8-95de-a4825e15e142" (UID: "e800cee6-7d63-49e8-95de-a4825e15e142"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:26.992049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.992020    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e800cee6-7d63-49e8-95de-a4825e15e142-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:26.992049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:26.992048    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfwl\" (UniqueName: \"kubernetes.io/projected/e800cee6-7d63-49e8-95de-a4825e15e142-kube-api-access-tsfwl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.136710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.136676    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"]
Oct 30 19:21:27.137442 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.137427    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e800cee6-7d63-49e8-95de-a4825e15e142" containerName="revoke-keys"
Oct 30 19:21:27.137489 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.137445    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e800cee6-7d63-49e8-95de-a4825e15e142" containerName="revoke-keys"
Oct 30 19:21:27.137565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.137556    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e800cee6-7d63-49e8-95de-a4825e15e142" containerName="revoke-keys"
Oct 30 19:21:27.140901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.140880    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.162780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.162752    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"]
Oct 30 19:21:27.168459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.168428    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"]
Oct 30 19:21:27.173189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.173168    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.214362 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.212641    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.214582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.212975    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.214658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.214629    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqgr\" (UniqueName: \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.219105 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.219064    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"]
Oct 30 19:21:27.316855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.316828    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.316971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.316873    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.316971 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.316944    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqgr\" (UniqueName: \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.317108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.316978    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hq5\" (UniqueName: \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.317108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.317070    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.317373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.317352    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.317841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.317815    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.319352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.319332    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.330625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.330603    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqgr\" (UniqueName: \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\") pod \"maas-api-revoke-sub-2e01bda1-7g5ff\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.420314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.420287    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.420454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.420355    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.420454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.420412    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hq5\" (UniqueName: \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.421311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.421285    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.423677 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.423653    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.440732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.440706    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hq5\" (UniqueName: \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\") pod \"maas-api-revoke-sub-0248f2d1-w745z\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.456518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.456494    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:27.520366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.520336    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:27.711840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.711746    5065 generic.go:356] "Generic (PLEG): container finished" podID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerID="f82fb8963a5bc6bdcaf6863460b5b9625c032803fa693c9154bf3efcac849ef6" exitCode=0
Oct 30 19:21:27.712011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.711863    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerDied","Data":"f82fb8963a5bc6bdcaf6863460b5b9625c032803fa693c9154bf3efcac849ef6"}
Oct 30 19:21:27.712011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.711926    5065 scope.go:122] "RemoveContainer" containerID="ac55f3d29a8d17ea79496afc7557052e0aeddfbe3b58b35d6e4d859d32da18cd"
Oct 30 19:21:27.717732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.716920    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69" event={"ID":"cebd859c-e39c-4350-a294-7f99ddcedc61","Type":"ContainerStarted","Data":"34cf35057ab2083945953386b1ff370244af0bc8e53e01d4a6976f41b41ea7e0"}
Oct 30 19:21:27.717732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.716980    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69" event={"ID":"cebd859c-e39c-4350-a294-7f99ddcedc61","Type":"ContainerStarted","Data":"6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5"}
Oct 30 19:21:27.720316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.720227    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n" event={"ID":"e800cee6-7d63-49e8-95de-a4825e15e142","Type":"ContainerDied","Data":"2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63"}
Oct 30 19:21:27.720316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.720259    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"
Oct 30 19:21:27.720316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.720273    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2217641aae2ed9cdc9a3d32ca6b7e7555d6a28a9949987d43fc38d1744963d63"
Oct 30 19:21:27.756517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.754302    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69" podStartSLOduration=1.754282759 podStartE2EDuration="1.754282759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:27.748567055 +0000 UTC m=+2908.012032032" watchObservedRunningTime="2026-10-30 19:21:27.754282759 +0000 UTC m=+2908.017747707"
Oct 30 19:21:27.781270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.779493    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"]
Oct 30 19:21:27.862418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.861276    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 19:21:27.927598 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.927574    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 19:21:27.927803 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.927792    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 19:21:27.962121 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:27.962043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:21:28.030775 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:28.030742    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf8b8c5_fe83_438d_9fb9_7aa65f780b6a.slice/crio-36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960 WatchSource:0}: Error finding container 36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960: Status 404 returned error can't find the container with id 36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960
Oct 30 19:21:28.034075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.034039    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"]
Oct 30 19:21:28.042004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.041973    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042108    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042199    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042224    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:28.042263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042251    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042324    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:28.042382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042355    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042442    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042455    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:28.042573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042546    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042632 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042605    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.042740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042719    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\") pod \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\" (UID: \"cdadd688-7c02-40e1-b665-2d3eaaff58fb\") "
Oct 30 19:21:28.043020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.042971    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:28.043175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.043147    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:28.044138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044111    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.044254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044142    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.044254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044159    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.044254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044174    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.044254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044188    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.044704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044677    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:28.044810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.044791    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:28.045034 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.045018    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:28.045326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.045308    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt" pod "cdadd688-7c02-40e1-b665-2d3eaaff58fb" (UID: "cdadd688-7c02-40e1-b665-2d3eaaff58fb"). InnerVolumeSpecName "kube-api-access-mjmnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:28.146173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.146135    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmnt\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-kube-api-access-mjmnt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.146173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.146173    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.146359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.146192    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.146359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.146207    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdadd688-7c02-40e1-b665-2d3eaaff58fb-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.731825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.731735    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"
Oct 30 19:21:28.731825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.731748    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw" event={"ID":"cdadd688-7c02-40e1-b665-2d3eaaff58fb","Type":"ContainerDied","Data":"1d911cbf9ecf670bb03342717df06092e6dd8e53a979ce8f30ab31ab24211411"}
Oct 30 19:21:28.731825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.731819    5065 scope.go:122] "RemoveContainer" containerID="f82fb8963a5bc6bdcaf6863460b5b9625c032803fa693c9154bf3efcac849ef6"
Oct 30 19:21:28.733630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.733606    5065 generic.go:356] "Generic (PLEG): container finished" podID="cebd859c-e39c-4350-a294-7f99ddcedc61" containerID="34cf35057ab2083945953386b1ff370244af0bc8e53e01d4a6976f41b41ea7e0" exitCode=0
Oct 30 19:21:28.733763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.733687    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69" event={"ID":"cebd859c-e39c-4350-a294-7f99ddcedc61","Type":"ContainerDied","Data":"34cf35057ab2083945953386b1ff370244af0bc8e53e01d4a6976f41b41ea7e0"}
Oct 30 19:21:28.735695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.735657    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z" event={"ID":"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a","Type":"ContainerStarted","Data":"fcc80dba6b26fc8645ff3d7c8c77e1ded201646e39746a0b7956c4041e5ab7de"}
Oct 30 19:21:28.735807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.735703    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z" event={"ID":"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a","Type":"ContainerStarted","Data":"36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960"}
Oct 30 19:21:28.737440 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.737418    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff" event={"ID":"08148bce-5390-4fbb-8327-e40781290654","Type":"ContainerStarted","Data":"74c0da0035984ae00995ae64d0da377ac6af9a7307edc27f29418bdc241b73cc"}
Oct 30 19:21:28.737543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.737454    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff" event={"ID":"08148bce-5390-4fbb-8327-e40781290654","Type":"ContainerStarted","Data":"765b293ccaff8d3500c22926f3ab5a2e2c94f091c11d73cc499a79d64bcc4bf8"}
Oct 30 19:21:28.787462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.777771    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z" podStartSLOduration=1.777750277 podStartE2EDuration="1.777750277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:28.767293249 +0000 UTC m=+2909.030758231" watchObservedRunningTime="2026-10-30 19:21:28.777750277 +0000 UTC m=+2909.041215226"
Oct 30 19:21:28.843341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.843291    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff" podStartSLOduration=1.843276673 podStartE2EDuration="1.843276673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:28.840579622 +0000 UTC m=+2909.104044572" watchObservedRunningTime="2026-10-30 19:21:28.843276673 +0000 UTC m=+2909.106741620"
Oct 30 19:21:28.874945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.874912    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 19:21:28.880741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:28.880713    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-xqtfw"]
Oct 30 19:21:29.761517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:29.761480    5065 generic.go:356] "Generic (PLEG): container finished" podID="08148bce-5390-4fbb-8327-e40781290654" containerID="74c0da0035984ae00995ae64d0da377ac6af9a7307edc27f29418bdc241b73cc" exitCode=0
Oct 30 19:21:29.761719 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:29.761572    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff" event={"ID":"08148bce-5390-4fbb-8327-e40781290654","Type":"ContainerDied","Data":"74c0da0035984ae00995ae64d0da377ac6af9a7307edc27f29418bdc241b73cc"}
Oct 30 19:21:29.764426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:29.764399    5065 generic.go:356] "Generic (PLEG): container finished" podID="aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" containerID="fcc80dba6b26fc8645ff3d7c8c77e1ded201646e39746a0b7956c4041e5ab7de" exitCode=0
Oct 30 19:21:29.764543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:29.764452    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z" event={"ID":"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a","Type":"ContainerDied","Data":"fcc80dba6b26fc8645ff3d7c8c77e1ded201646e39746a0b7956c4041e5ab7de"}
Oct 30 19:21:30.030657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.030584    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" path="/var/lib/kubelet/pods/cdadd688-7c02-40e1-b665-2d3eaaff58fb/volumes"
Oct 30 19:21:30.031779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.031077    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"]
Oct 30 19:21:30.032565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.032539    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:30.032678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.032567    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:30.032678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.032611    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:30.032678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.032620    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:30.032834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.032814    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:30.038527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.038500    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"]
Oct 30 19:21:30.038650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.038614    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.046512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.046489    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:30.129190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.129149    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\") pod \"cebd859c-e39c-4350-a294-7f99ddcedc61\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") "
Oct 30 19:21:30.129367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.129352    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\" (UniqueName: \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\") pod \"cebd859c-e39c-4350-a294-7f99ddcedc61\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") "
Oct 30 19:21:30.129452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.129435    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\") pod \"cebd859c-e39c-4350-a294-7f99ddcedc61\" (UID: \"cebd859c-e39c-4350-a294-7f99ddcedc61\") "
Oct 30 19:21:30.129569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.129542    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca" pod "cebd859c-e39c-4350-a294-7f99ddcedc61" (UID: "cebd859c-e39c-4350-a294-7f99ddcedc61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:30.130058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.129978    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwf52\" (UniqueName: \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.130058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.130022    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.130357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.130194    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.130357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.130340    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.131827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.131803    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75" pod "cebd859c-e39c-4350-a294-7f99ddcedc61" (UID: "cebd859c-e39c-4350-a294-7f99ddcedc61"). InnerVolumeSpecName "kube-api-access-b6w75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:30.132294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.132273    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca" pod "cebd859c-e39c-4350-a294-7f99ddcedc61" (UID: "cebd859c-e39c-4350-a294-7f99ddcedc61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:30.232025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.231990    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwf52\" (UniqueName: \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.232161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.232039    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.232161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.232132    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.232299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.232281    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6w75\" (UniqueName: \"kubernetes.io/projected/cebd859c-e39c-4350-a294-7f99ddcedc61-kube-api-access-b6w75\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.232351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.232301    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebd859c-e39c-4350-a294-7f99ddcedc61-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:30.232680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.232652    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.234600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.234580    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.246023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.246004    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwf52\" (UniqueName: \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.358931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.358868    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:30.652922 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:30.652894    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44b913c_e101_4e3c_bda5_cc3a2471c014.slice/crio-34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a WatchSource:0}: Error finding container 34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a: Status 404 returned error can't find the container with id 34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a
Oct 30 19:21:30.654798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.654772    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"]
Oct 30 19:21:30.779234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.779210    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"
Oct 30 19:21:30.779367 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.779204    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69" event={"ID":"cebd859c-e39c-4350-a294-7f99ddcedc61","Type":"ContainerDied","Data":"6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5"}
Oct 30 19:21:30.779458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.779395    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdedc1330a207f23e3cd1f4fcbae1093152813984c42f1cd2369c4fd41be2e5"
Oct 30 19:21:30.781221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:30.781197    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7" event={"ID":"a44b913c-e101-4e3c-bda5-cc3a2471c014","Type":"ContainerStarted","Data":"34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a"}
Oct 30 19:21:31.130130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.129522    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:31.203624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.203556    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:31.251516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251487    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\" (UniqueName: \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\") pod \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") "
Oct 30 19:21:31.251689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251571    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\") pod \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") "
Oct 30 19:21:31.251689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251605    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\") pod \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\" (UID: \"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a\") "
Oct 30 19:21:31.251689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251656    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\") pod \"08148bce-5390-4fbb-8327-e40781290654\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") "
Oct 30 19:21:31.251938 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251909    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\") pod \"08148bce-5390-4fbb-8327-e40781290654\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") "
Oct 30 19:21:31.252011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251948    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca" pod "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" (UID: "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:31.252011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.251982    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\" (UniqueName: \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\") pod \"08148bce-5390-4fbb-8327-e40781290654\" (UID: \"08148bce-5390-4fbb-8327-e40781290654\") "
Oct 30 19:21:31.252387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.252327    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca" pod "08148bce-5390-4fbb-8327-e40781290654" (UID: "08148bce-5390-4fbb-8327-e40781290654"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:31.252917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.252889    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08148bce-5390-4fbb-8327-e40781290654-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.253012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.252922    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.254370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.254287    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca" pod "08148bce-5390-4fbb-8327-e40781290654" (UID: "08148bce-5390-4fbb-8327-e40781290654"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:31.254840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.254806    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca" pod "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" (UID: "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:31.254944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.254892    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr" pod "08148bce-5390-4fbb-8327-e40781290654" (UID: "08148bce-5390-4fbb-8327-e40781290654"). InnerVolumeSpecName "kube-api-access-wdqgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:31.255438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.255415    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5" pod "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" (UID: "aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a"). InnerVolumeSpecName "kube-api-access-r6hq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:31.354408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.354381    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08148bce-5390-4fbb-8327-e40781290654-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.354408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.354408    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdqgr\" (UniqueName: \"kubernetes.io/projected/08148bce-5390-4fbb-8327-e40781290654-kube-api-access-wdqgr\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.354603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.354418    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hq5\" (UniqueName: \"kubernetes.io/projected/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-kube-api-access-r6hq5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.354603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.354428    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:31.793841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.793803    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff" event={"ID":"08148bce-5390-4fbb-8327-e40781290654","Type":"ContainerDied","Data":"765b293ccaff8d3500c22926f3ab5a2e2c94f091c11d73cc499a79d64bcc4bf8"}
Oct 30 19:21:31.794043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.793892    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="765b293ccaff8d3500c22926f3ab5a2e2c94f091c11d73cc499a79d64bcc4bf8"
Oct 30 19:21:31.794043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.793802    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"
Oct 30 19:21:31.795798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.795761    5065 generic.go:356] "Generic (PLEG): container finished" podID="a44b913c-e101-4e3c-bda5-cc3a2471c014" containerID="6e0476e4e22479f8f946d6592712e799b5b1a2ed263e51dedfeedf5d4d850e49" exitCode=0
Oct 30 19:21:31.795964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.795935    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7" event={"ID":"a44b913c-e101-4e3c-bda5-cc3a2471c014","Type":"ContainerDied","Data":"6e0476e4e22479f8f946d6592712e799b5b1a2ed263e51dedfeedf5d4d850e49"}
Oct 30 19:21:31.797615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.797595    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"
Oct 30 19:21:31.797768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.797626    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z" event={"ID":"aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a","Type":"ContainerDied","Data":"36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960"}
Oct 30 19:21:31.797768 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:31.797666    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36025ae2a3ad80adf4ef7070232deec3b8d029372b6046bb2f8672ac1e6e0960"
Oct 30 19:21:32.047593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:32.047555    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:32.047780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:32.047590    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:33.057218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.057188    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:33.189174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.189148    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\") pod \"a44b913c-e101-4e3c-bda5-cc3a2471c014\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") "
Oct 30 19:21:33.189351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.189338    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\") pod \"a44b913c-e101-4e3c-bda5-cc3a2471c014\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") "
Oct 30 19:21:33.189438 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.189403    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\" (UniqueName: \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\") pod \"a44b913c-e101-4e3c-bda5-cc3a2471c014\" (UID: \"a44b913c-e101-4e3c-bda5-cc3a2471c014\") "
Oct 30 19:21:33.189838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.189811    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca" pod "a44b913c-e101-4e3c-bda5-cc3a2471c014" (UID: "a44b913c-e101-4e3c-bda5-cc3a2471c014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:33.189979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.189944    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.192223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.192191    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca" pod "a44b913c-e101-4e3c-bda5-cc3a2471c014" (UID: "a44b913c-e101-4e3c-bda5-cc3a2471c014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:33.192390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.192369    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52" pod "a44b913c-e101-4e3c-bda5-cc3a2471c014" (UID: "a44b913c-e101-4e3c-bda5-cc3a2471c014"). InnerVolumeSpecName "kube-api-access-rwf52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:33.292743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.292385    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44b913c-e101-4e3c-bda5-cc3a2471c014-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.292743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.292421    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwf52\" (UniqueName: \"kubernetes.io/projected/a44b913c-e101-4e3c-bda5-cc3a2471c014-kube-api-access-rwf52\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.415113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.407076    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 19:21:33.423227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.423195    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 19:21:33.442013 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.440422    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 19:21:33.478522 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.478490    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 19:21:33.494789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.494733    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\" (UniqueName: \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\") pod \"347f1c25-bf10-4847-a907-8ccfad8a668f\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") "
Oct 30 19:21:33.494975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.494885    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\") pod \"347f1c25-bf10-4847-a907-8ccfad8a668f\" (UID: \"347f1c25-bf10-4847-a907-8ccfad8a668f\") "
Oct 30 19:21:33.496835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.496801    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume" pod "347f1c25-bf10-4847-a907-8ccfad8a668f" (UID: "347f1c25-bf10-4847-a907-8ccfad8a668f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:33.501766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.501737    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk" pod "347f1c25-bf10-4847-a907-8ccfad8a668f" (UID: "347f1c25-bf10-4847-a907-8ccfad8a668f"). InnerVolumeSpecName "kube-api-access-l5svk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:33.597308 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.597265    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\") pod \"fc5cbdda-fba6-4c54-b196-121d410d97c8\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") "
Oct 30 19:21:33.597572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.597550    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\" (UniqueName: \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\") pod \"fc5cbdda-fba6-4c54-b196-121d410d97c8\" (UID: \"fc5cbdda-fba6-4c54-b196-121d410d97c8\") "
Oct 30 19:21:33.598547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.598517    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5svk\" (UniqueName: \"kubernetes.io/projected/347f1c25-bf10-4847-a907-8ccfad8a668f-kube-api-access-l5svk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.598679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.598562    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347f1c25-bf10-4847-a907-8ccfad8a668f-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.599620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.599594    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume" pod "fc5cbdda-fba6-4c54-b196-121d410d97c8" (UID: "fc5cbdda-fba6-4c54-b196-121d410d97c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:33.601503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.601477    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg" pod "fc5cbdda-fba6-4c54-b196-121d410d97c8" (UID: "fc5cbdda-fba6-4c54-b196-121d410d97c8"). InnerVolumeSpecName "kube-api-access-fs6dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:33.711181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.703800    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6dg\" (UniqueName: \"kubernetes.io/projected/fc5cbdda-fba6-4c54-b196-121d410d97c8-kube-api-access-fs6dg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.711181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.703840    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cbdda-fba6-4c54-b196-121d410d97c8-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:33.910961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.910931    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"
Oct 30 19:21:33.928875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.928840    5065 generic.go:356] "Generic (PLEG): container finished" podID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerID="f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4" exitCode=0
Oct 30 19:21:33.929267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.929249    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"
Oct 30 19:21:33.943126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.942845    5065 generic.go:356] "Generic (PLEG): container finished" podID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerID="a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d" exitCode=0
Oct 30 19:21:33.943126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.943079    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981683    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7" event={"ID":"a44b913c-e101-4e3c-bda5-cc3a2471c014","Type":"ContainerDied","Data":"34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a"}
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981717    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a989f430026ecaecd76bb1d58eeb8d9624c019cab07cfd53c6033604a0954a"
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981732    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" event={"ID":"347f1c25-bf10-4847-a907-8ccfad8a668f","Type":"ContainerDied","Data":"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"}
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981747    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb" event={"ID":"347f1c25-bf10-4847-a907-8ccfad8a668f","Type":"ContainerDied","Data":"cb9a08eb179ceed98931b4946ec53194dd24da42ee891200426a274efbe772a6"}
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981761    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" event={"ID":"fc5cbdda-fba6-4c54-b196-121d410d97c8","Type":"ContainerDied","Data":"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"}
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981775    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls" event={"ID":"fc5cbdda-fba6-4c54-b196-121d410d97c8","Type":"ContainerDied","Data":"efd2991dfbf13cda1ca0bf5210fe7e713e3abd4a8f5b809763b19745a9d530f1"}
Oct 30 19:21:33.982670 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.981795    5065 scope.go:122] "RemoveContainer" containerID="f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"
Oct 30 19:21:33.994144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:33.993538    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 19:21:34.002123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.001268    5065 scope.go:122] "RemoveContainer" containerID="f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"
Oct 30 19:21:34.002123 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:34.001558    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4\": container with ID starting with f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4 not found: ID does not exist" containerID="f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"
Oct 30 19:21:34.002123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.001590    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4"} err="failed to get container status \"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4\": rpc error: code = NotFound desc = could not find container \"f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4\": container with ID starting with f8164124bf73a3e97dfca6e6bf4eccf9a8c5a2dbfae4dca403ba720cba08d1e4 not found: ID does not exist"
Oct 30 19:21:34.002123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.001612    5065 scope.go:122] "RemoveContainer" containerID="a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"
Oct 30 19:21:34.006466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.006441    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-304f22-b5cf775c8-m52mb"]
Oct 30 19:21:34.015629 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:34.015604    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44b913c_e101_4e3c_bda5_cc3a2471c014.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:21:34.019104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.019064    5065 scope.go:122] "RemoveContainer" containerID="a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"
Oct 30 19:21:34.019540 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:34.019492    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d\": container with ID starting with a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d not found: ID does not exist" containerID="a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"
Oct 30 19:21:34.019540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.019518    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d"} err="failed to get container status \"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d\": rpc error: code = NotFound desc = could not find container \"a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d\": container with ID starting with a26dfab3cdaaf20e00ddd433c68d282ce97cb2856bb972efa05e078ad7b0d68d not found: ID does not exist"
Oct 30 19:21:34.025902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.025877    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 19:21:34.031053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:34.031034    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-304f22-646d77d859-nmwls"]
Oct 30 19:21:35.897448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:35.897403    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" path="/var/lib/kubelet/pods/347f1c25-bf10-4847-a907-8ccfad8a668f/volumes"
Oct 30 19:21:35.901080 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:35.901045    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" path="/var/lib/kubelet/pods/fc5cbdda-fba6-4c54-b196-121d410d97c8/volumes"
Oct 30 19:21:37.990531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:37.990493    5065 generic.go:356] "Generic (PLEG): container finished" podID="565da0d2-ac21-468f-aee9-af8467737307" containerID="7b507b7880e76f7c400445d8683e20c9511e038865272893cf70fe9c3ffd1a80" exitCode=0
Oct 30 19:21:37.991355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:37.990644    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" event={"ID":"565da0d2-ac21-468f-aee9-af8467737307","Type":"ContainerDied","Data":"7b507b7880e76f7c400445d8683e20c9511e038865272893cf70fe9c3ffd1a80"}
Oct 30 19:21:37.992955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:37.992930    5065 generic.go:356] "Generic (PLEG): container finished" podID="94bee4cb-9318-42b1-b336-f664037b57e2" containerID="d82c7f9f774ed0f309e4b74b8f06e0a6f0a630203e2f556ab9cfe3b0c3e17f99" exitCode=0
Oct 30 19:21:37.993134 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:37.993075    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" event={"ID":"94bee4cb-9318-42b1-b336-f664037b57e2","Type":"ContainerDied","Data":"d82c7f9f774ed0f309e4b74b8f06e0a6f0a630203e2f556ab9cfe3b0c3e17f99"}
Oct 30 19:21:38.342898 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.342855    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"]
Oct 30 19:21:38.344259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344230    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a44b913c-e101-4e3c-bda5-cc3a2471c014" containerName="revoke-keys"
Oct 30 19:21:38.344435 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344421    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44b913c-e101-4e3c-bda5-cc3a2471c014" containerName="revoke-keys"
Oct 30 19:21:38.344575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344565    5065 cpu_manager.go:414] "containerMap: removing container" podUID="08148bce-5390-4fbb-8327-e40781290654" containerName="revoke-keys"
Oct 30 19:21:38.344664 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344653    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08148bce-5390-4fbb-8327-e40781290654" containerName="revoke-keys"
Oct 30 19:21:38.344747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344737    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing"
Oct 30 19:21:38.344827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344816    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing"
Oct 30 19:21:38.344932 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.344922    5065 cpu_manager.go:414] "containerMap: removing container" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing"
Oct 30 19:21:38.345012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345002    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing"
Oct 30 19:21:38.345130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345117    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cebd859c-e39c-4350-a294-7f99ddcedc61" containerName="revoke-keys"
Oct 30 19:21:38.345220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345210    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cebd859c-e39c-4350-a294-7f99ddcedc61" containerName="revoke-keys"
Oct 30 19:21:38.345317 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345307    5065 cpu_manager.go:414] "containerMap: removing container" podUID="aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" containerName="revoke-keys"
Oct 30 19:21:38.345398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345389    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" containerName="revoke-keys"
Oct 30 19:21:38.345650 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345638    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" containerName="revoke-keys"
Oct 30 19:21:38.345750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345740    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44b913c-e101-4e3c-bda5-cc3a2471c014" containerName="revoke-keys"
Oct 30 19:21:38.345845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345834    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="08148bce-5390-4fbb-8327-e40781290654" containerName="revoke-keys"
Oct 30 19:21:38.345933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.345924    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5cbdda-fba6-4c54-b196-121d410d97c8" containerName="payload-pre-processing"
Oct 30 19:21:38.346014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.346005    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdadd688-7c02-40e1-b665-2d3eaaff58fb" containerName="istio-proxy"
Oct 30 19:21:38.346122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.346111    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="347f1c25-bf10-4847-a907-8ccfad8a668f" containerName="payload-processing"
Oct 30 19:21:38.346225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.346216    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cebd859c-e39c-4350-a294-7f99ddcedc61" containerName="revoke-keys"
Oct 30 19:21:38.354106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.351157    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"]
Oct 30 19:21:38.354106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.351196    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"]
Oct 30 19:21:38.356138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.354413    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.357504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.357485    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.360966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.360947    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:38.374880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.361814    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:38.374880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.374627    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"]
Oct 30 19:21:38.424494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.424453    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing" containerID="cri-o://68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d" gracePeriod=25
Oct 30 19:21:38.441218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.441179    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing" containerID="cri-o://543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06" gracePeriod=25
Oct 30 19:21:38.462252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.462228    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.488900    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.488983    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.489044    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kb7\" (UniqueName: \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.489106    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5qm\" (UniqueName: \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.489137    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.492146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.491681    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.533459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.533432    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 19:21:38.593155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593040    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\") pod \"94bee4cb-9318-42b1-b336-f664037b57e2\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") "
Oct 30 19:21:38.593155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593124    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\" (UniqueName: \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\") pod \"94bee4cb-9318-42b1-b336-f664037b57e2\" (UID: \"94bee4cb-9318-42b1-b336-f664037b57e2\") "
Oct 30 19:21:38.593478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593455    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\") pod \"565da0d2-ac21-468f-aee9-af8467737307\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") "
Oct 30 19:21:38.593807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593787    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.593920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593858    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.593980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593925    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kb7\" (UniqueName: \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.594043 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.593973    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume" pod "565da0d2-ac21-468f-aee9-af8467737307" (UID: "565da0d2-ac21-468f-aee9-af8467737307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:38.594195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.594174    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5qm\" (UniqueName: \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.594264 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.594246    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.594346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.594330    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.594928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.594634    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565da0d2-ac21-468f-aee9-af8467737307-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:38.595427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.595401    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.595771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.595748    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.595909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.595884    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume" pod "94bee4cb-9318-42b1-b336-f664037b57e2" (UID: "94bee4cb-9318-42b1-b336-f664037b57e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:38.596037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.595961    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs" pod "94bee4cb-9318-42b1-b336-f664037b57e2" (UID: "94bee4cb-9318-42b1-b336-f664037b57e2"). InnerVolumeSpecName "kube-api-access-zgbjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:38.597127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.597080    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.597978 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.597961    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.609573 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.609545    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5qm\" (UniqueName: \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\") pod \"maas-api-revoke-sub-96a84ebc-x5v89\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.609837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.609807    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kb7\" (UniqueName: \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\") pod \"maas-api-revoke-sub-3d29fe69-qzlfc\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.714110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.695476    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\" (UniqueName: \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\") pod \"565da0d2-ac21-468f-aee9-af8467737307\" (UID: \"565da0d2-ac21-468f-aee9-af8467737307\") "
Oct 30 19:21:38.714110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.699200    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bee4cb-9318-42b1-b336-f664037b57e2-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:38.714110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.699230    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbjs\" (UniqueName: \"kubernetes.io/projected/94bee4cb-9318-42b1-b336-f664037b57e2-kube-api-access-zgbjs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:38.716248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.715820    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:38.720106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.716575    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:38.798198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.798101    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm" pod "565da0d2-ac21-468f-aee9-af8467737307" (UID: "565da0d2-ac21-468f-aee9-af8467737307"). InnerVolumeSpecName "kube-api-access-dhcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:38.814652 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:38.814614    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcbm\" (UniqueName: \"kubernetes.io/projected/565da0d2-ac21-468f-aee9-af8467737307-kube-api-access-dhcbm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:39.007909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.007852    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj" event={"ID":"565da0d2-ac21-468f-aee9-af8467737307","Type":"ContainerDied","Data":"9e5fc20bc6420b0a66a11282cb67adc372c752b9084e4c752dbc5b8f318d7f92"}
Oct 30 19:21:39.009011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.007933    5065 scope.go:122] "RemoveContainer" containerID="7b507b7880e76f7c400445d8683e20c9511e038865272893cf70fe9c3ffd1a80"
Oct 30 19:21:39.009011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.008133    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"
Oct 30 19:21:39.015566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.015415    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"
Oct 30 19:21:39.015566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.015491    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489" event={"ID":"94bee4cb-9318-42b1-b336-f664037b57e2","Type":"ContainerDied","Data":"a457a66867576ea0b9298ee7bec5e9ef1c72279b7ecece8f21f1051f59ef732a"}
Oct 30 19:21:39.038528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.038502    5065 scope.go:122] "RemoveContainer" containerID="d82c7f9f774ed0f309e4b74b8f06e0a6f0a630203e2f556ab9cfe3b0c3e17f99"
Oct 30 19:21:39.049944 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.049920    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 19:21:39.061501 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.061453    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-aa6736-76ffb896f5-7nrqj"]
Oct 30 19:21:39.073190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.073169    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 19:21:39.078547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.078527    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-aa6736-95dbf4544-zs489"]
Oct 30 19:21:39.145406 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:39.145375    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b01866_a6da_4939_9bdb_83d970b07ef6.slice/crio-d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68 WatchSource:0}: Error finding container d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68: Status 404 returned error can't find the container with id d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68
Oct 30 19:21:39.145543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.145486    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"]
Oct 30 19:21:39.174303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.174271    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"]
Oct 30 19:21:39.959807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.959672    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565da0d2-ac21-468f-aee9-af8467737307" path="/var/lib/kubelet/pods/565da0d2-ac21-468f-aee9-af8467737307/volumes"
Oct 30 19:21:39.960709 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:39.960684    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" path="/var/lib/kubelet/pods/94bee4cb-9318-42b1-b336-f664037b57e2/volumes"
Oct 30 19:21:40.033637 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.033602    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89" event={"ID":"621023c3-d46d-48ca-9e22-dc7ce4505af0","Type":"ContainerStarted","Data":"f427c0d6c4fcc91e0a421259e54fece6de5436e59a8d1ccc2f7e2e3b4c82393d"}
Oct 30 19:21:40.034143 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.033660    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89" event={"ID":"621023c3-d46d-48ca-9e22-dc7ce4505af0","Type":"ContainerStarted","Data":"f7ed7669f6d74d2b250820650f9bc3e1615b8ac5ed1895c59b5672e1b44befbd"}
Oct 30 19:21:40.035468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.035445    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc" event={"ID":"63b01866-a6da-4939-9bdb-83d970b07ef6","Type":"ContainerStarted","Data":"7d65589b1318bc5ab764cfd228bb1e568135469719400833a99cfd2d2ce71962"}
Oct 30 19:21:40.035590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.035485    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc" event={"ID":"63b01866-a6da-4939-9bdb-83d970b07ef6","Type":"ContainerStarted","Data":"d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68"}
Oct 30 19:21:40.064646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.064591    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89" podStartSLOduration=2.064575138 podStartE2EDuration="2.064575138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:40.061844678 +0000 UTC m=+2920.325309646" watchObservedRunningTime="2026-10-30 19:21:40.064575138 +0000 UTC m=+2920.328040088"
Oct 30 19:21:40.089531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.089491    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc" podStartSLOduration=2.089478939 podStartE2EDuration="2.089478939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:40.085501387 +0000 UTC m=+2920.348966356" watchObservedRunningTime="2026-10-30 19:21:40.089478939 +0000 UTC m=+2920.352943924"
Oct 30 19:21:40.827882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.827803    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"]
Oct 30 19:21:40.843066 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:40.843010    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-aa6736-6c7bb"]
Oct 30 19:21:41.045911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.045878    5065 generic.go:356] "Generic (PLEG): container finished" podID="63b01866-a6da-4939-9bdb-83d970b07ef6" containerID="7d65589b1318bc5ab764cfd228bb1e568135469719400833a99cfd2d2ce71962" exitCode=0
Oct 30 19:21:41.046418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.045965    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc" event={"ID":"63b01866-a6da-4939-9bdb-83d970b07ef6","Type":"ContainerDied","Data":"7d65589b1318bc5ab764cfd228bb1e568135469719400833a99cfd2d2ce71962"}
Oct 30 19:21:41.047910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.047885    5065 generic.go:356] "Generic (PLEG): container finished" podID="621023c3-d46d-48ca-9e22-dc7ce4505af0" containerID="f427c0d6c4fcc91e0a421259e54fece6de5436e59a8d1ccc2f7e2e3b4c82393d" exitCode=0
Oct 30 19:21:41.048044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.047929    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89" event={"ID":"621023c3-d46d-48ca-9e22-dc7ce4505af0","Type":"ContainerDied","Data":"f427c0d6c4fcc91e0a421259e54fece6de5436e59a8d1ccc2f7e2e3b4c82393d"}
Oct 30 19:21:41.953642 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.953602    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:41.954203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.954181    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:41.954296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:41.954214    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65908435-5699-4ea0-ac57-e39865463239" path="/var/lib/kubelet/pods/65908435-5699-4ea0-ac57-e39865463239/volumes"
Oct 30 19:21:42.281109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.281072    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:42.283842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.283820    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:42.421387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421354    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\" (UniqueName: \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\") pod \"621023c3-d46d-48ca-9e22-dc7ce4505af0\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") "
Oct 30 19:21:42.421577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421478    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\") pod \"63b01866-a6da-4939-9bdb-83d970b07ef6\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") "
Oct 30 19:21:42.421577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421550    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\") pod \"63b01866-a6da-4939-9bdb-83d970b07ef6\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") "
Oct 30 19:21:42.421718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421696    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\") pod \"621023c3-d46d-48ca-9e22-dc7ce4505af0\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") "
Oct 30 19:21:42.422126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421804    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\") pod \"621023c3-d46d-48ca-9e22-dc7ce4505af0\" (UID: \"621023c3-d46d-48ca-9e22-dc7ce4505af0\") "
Oct 30 19:21:42.422126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421847    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca" pod "63b01866-a6da-4939-9bdb-83d970b07ef6" (UID: "63b01866-a6da-4939-9bdb-83d970b07ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:42.422126 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.421891    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\" (UniqueName: \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\") pod \"63b01866-a6da-4939-9bdb-83d970b07ef6\" (UID: \"63b01866-a6da-4939-9bdb-83d970b07ef6\") "
Oct 30 19:21:42.422357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.422208    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca" pod "621023c3-d46d-48ca-9e22-dc7ce4505af0" (UID: "621023c3-d46d-48ca-9e22-dc7ce4505af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:42.422572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.422544    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.422647 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.422574    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.424914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.424887    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7" pod "63b01866-a6da-4939-9bdb-83d970b07ef6" (UID: "63b01866-a6da-4939-9bdb-83d970b07ef6"). InnerVolumeSpecName "kube-api-access-d5kb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:42.425039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.424926    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm" pod "621023c3-d46d-48ca-9e22-dc7ce4505af0" (UID: "621023c3-d46d-48ca-9e22-dc7ce4505af0"). InnerVolumeSpecName "kube-api-access-nz5qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:42.425525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.425499    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca" pod "621023c3-d46d-48ca-9e22-dc7ce4505af0" (UID: "621023c3-d46d-48ca-9e22-dc7ce4505af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:42.425869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.425843    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca" pod "63b01866-a6da-4939-9bdb-83d970b07ef6" (UID: "63b01866-a6da-4939-9bdb-83d970b07ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:42.524342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.524311    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621023c3-d46d-48ca-9e22-dc7ce4505af0-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.524484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.524347    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kb7\" (UniqueName: \"kubernetes.io/projected/63b01866-a6da-4939-9bdb-83d970b07ef6-kube-api-access-d5kb7\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.524484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.524363    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz5qm\" (UniqueName: \"kubernetes.io/projected/621023c3-d46d-48ca-9e22-dc7ce4505af0-kube-api-access-nz5qm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.524484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.524377    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b01866-a6da-4939-9bdb-83d970b07ef6-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:42.935132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.935023    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main" containerID="cri-o://5ce40812f6ee2ae0251b5153ada52b6bbb80e740b71622c8910df75ac2398673" gracePeriod=45
Oct 30 19:21:42.961244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.961216    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main" containerID="cri-o://1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762" gracePeriod=45
Oct 30 19:21:42.964296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:42.964240    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main" containerID="cri-o://36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598" gracePeriod=45
Oct 30 19:21:43.065818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.065792    5065 generic.go:356] "Generic (PLEG): container finished" podID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerID="5ce40812f6ee2ae0251b5153ada52b6bbb80e740b71622c8910df75ac2398673" exitCode=0
Oct 30 19:21:43.065915 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.065860    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" event={"ID":"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5","Type":"ContainerDied","Data":"5ce40812f6ee2ae0251b5153ada52b6bbb80e740b71622c8910df75ac2398673"}
Oct 30 19:21:43.067514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.067495    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"
Oct 30 19:21:43.067619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.067512    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc" event={"ID":"63b01866-a6da-4939-9bdb-83d970b07ef6","Type":"ContainerDied","Data":"d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68"}
Oct 30 19:21:43.067619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.067562    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7953e29475dcc8385bbf4545acf89b1f63c31a9e49bd2a5cc08583b3ae6ea68"
Oct 30 19:21:43.069402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.069375    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89" event={"ID":"621023c3-d46d-48ca-9e22-dc7ce4505af0","Type":"ContainerDied","Data":"f7ed7669f6d74d2b250820650f9bc3e1615b8ac5ed1895c59b5672e1b44befbd"}
Oct 30 19:21:43.069402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.069399    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"
Oct 30 19:21:43.069925 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.069424    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7ed7669f6d74d2b250820650f9bc3e1615b8ac5ed1895c59b5672e1b44befbd"
Oct 30 19:21:43.272953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.272929    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338474    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338561    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338755    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338874    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\" (UniqueName: \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338921    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.339780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.338971    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\") pod \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\" (UID: \"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5\") "
Oct 30 19:21:43.341138 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.341072    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.341365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.341329    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.341667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.341624    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.345419 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.345270    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.347000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.346975    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:43.347355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.347322    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh" pod "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" (UID: "a4f78b43-62f9-4fee-851a-8bf6d8cf70e5"). InnerVolumeSpecName "kube-api-access-mqqjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:43.371737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.371711    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:21:43.373683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.373662    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:21:43.440712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.440600    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\" (UniqueName: \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.440954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.440796    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.440954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.440890    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.441205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.440966    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.441205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441024    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\" (UniqueName: \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.441205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441079    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.441205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441171    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\" (UniqueName: \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.441484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441237    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.441484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441329    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.441586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441498    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.441586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441561    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\") pod \"e3e44f70-dd7c-412a-be3d-6a2260040484\" (UID: \"e3e44f70-dd7c-412a-be3d-6a2260040484\") "
Oct 30 19:21:43.441678 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.441591    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\") pod \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\" (UID: \"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a\") "
Oct 30 19:21:43.442268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442242    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442271    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqjh\" (UniqueName: \"kubernetes.io/projected/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-kube-api-access-mqqjh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442287    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442300    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442312    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442323    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.442635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.442570    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.443587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.443565    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.443824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.443592    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.443979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.443953    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.444069 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.443741    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.445504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.445480    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.448168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.448132    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.448370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.448349    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "kube-api-access-vmp58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:43.449931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.449900    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:43.452510 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.452480    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "kube-api-access-5s4sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:43.453633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.453606    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm" pod "e3e44f70-dd7c-412a-be3d-6a2260040484" (UID: "e3e44f70-dd7c-412a-be3d-6a2260040484"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.453952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.453925    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs" pod "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" (UID: "556e9d5c-0c3c-4ae8-8c7f-435d9a95120a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544120    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e44f70-dd7c-412a-be3d-6a2260040484-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544155    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544169    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544182    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544195    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544208    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544221    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4sp\" (UniqueName: \"kubernetes.io/projected/e3e44f70-dd7c-412a-be3d-6a2260040484-kube-api-access-5s4sp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544236    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544250    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmp58\" (UniqueName: \"kubernetes.io/projected/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-kube-api-access-vmp58\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.544249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544264    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.545430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544278    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.545430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:43.544294    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e44f70-dd7c-412a-be3d-6a2260040484-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:44.079690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.079652    5065 generic.go:356] "Generic (PLEG): container finished" podID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerID="1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762" exitCode=0
Oct 30 19:21:44.079870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.079722    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"
Oct 30 19:21:44.079870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.079742    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" event={"ID":"e3e44f70-dd7c-412a-be3d-6a2260040484","Type":"ContainerDied","Data":"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"}
Oct 30 19:21:44.079870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.079794    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq" event={"ID":"e3e44f70-dd7c-412a-be3d-6a2260040484","Type":"ContainerDied","Data":"53eddc2dc6c5b5b910831d70f2ad737fb403ef03a6f24e7ec4ceefc87ceeb154"}
Oct 30 19:21:44.079870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.079817    5065 scope.go:122] "RemoveContainer" containerID="1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"
Oct 30 19:21:44.081837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.081786    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"
Oct 30 19:21:44.081837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.081861    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr" event={"ID":"a4f78b43-62f9-4fee-851a-8bf6d8cf70e5","Type":"ContainerDied","Data":"54fb958065bd1f8416f33681041a4c8b9cdfc0b2f1296fd2138754df7756af11"}
Oct 30 19:21:44.083721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.083687    5065 generic.go:356] "Generic (PLEG): container finished" podID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerID="36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598" exitCode=0
Oct 30 19:21:44.083937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.083735    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" event={"ID":"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a","Type":"ContainerDied","Data":"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"}
Oct 30 19:21:44.084062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.083959    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9" event={"ID":"556e9d5c-0c3c-4ae8-8c7f-435d9a95120a","Type":"ContainerDied","Data":"1daab8f6222165f9f97787d4b602207d732f98b22a0ae943fc82ae61d381890c"}
Oct 30 19:21:44.084062 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.083744    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"
Oct 30 19:21:44.094679 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.094661    5065 scope.go:122] "RemoveContainer" containerID="1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"
Oct 30 19:21:44.094962 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:44.094944    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762\": container with ID starting with 1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762 not found: ID does not exist" containerID="1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"
Oct 30 19:21:44.095044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.094976    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762"} err="failed to get container status \"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762\": rpc error: code = NotFound desc = could not find container \"1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762\": container with ID starting with 1f39c8f3b3580cf7fac6ac8bc6a2928e3baa1b486b20738122d98c24fc5e1762 not found: ID does not exist"
Oct 30 19:21:44.095044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.094993    5065 scope.go:122] "RemoveContainer" containerID="5ce40812f6ee2ae0251b5153ada52b6bbb80e740b71622c8910df75ac2398673"
Oct 30 19:21:44.108195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.108175    5065 scope.go:122] "RemoveContainer" containerID="36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"
Oct 30 19:21:44.118532 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.118508    5065 scope.go:122] "RemoveContainer" containerID="36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"
Oct 30 19:21:44.118801 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:44.118782    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598\": container with ID starting with 36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598 not found: ID does not exist" containerID="36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"
Oct 30 19:21:44.118862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.118816    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598"} err="failed to get container status \"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598\": rpc error: code = NotFound desc = could not find container \"36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598\": container with ID starting with 36ba815151916836e539628e31b0bc08cf9bd225efc3869e8c43fb5412fce598 not found: ID does not exist"
Oct 30 19:21:44.146958 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.146937    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 19:21:44.159718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.159664    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-2-w4-da70a5-kserve-5778fdd7c6-cvmfr"]
Oct 30 19:21:44.240891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.240864    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 19:21:44.266326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.266306    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/simulator-w4-da70a5-kserve-75dc4d8589-4h4dq"]
Oct 30 19:21:44.290228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.290205    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 19:21:44.297651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:44.297629    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/premium-w4-da70a5-kserve-5959dbc688-hc2d9"]
Oct 30 19:21:45.896350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:45.896314    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" path="/var/lib/kubelet/pods/556e9d5c-0c3c-4ae8-8c7f-435d9a95120a/volumes"
Oct 30 19:21:45.896997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:45.896975    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" path="/var/lib/kubelet/pods/a4f78b43-62f9-4fee-851a-8bf6d8cf70e5/volumes"
Oct 30 19:21:45.897594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:45.897574    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" path="/var/lib/kubelet/pods/e3e44f70-dd7c-412a-be3d-6a2260040484/volumes"
Oct 30 19:21:47.892828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:47.892796    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"]
Oct 30 19:21:47.892828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:47.892832    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6d7870-9zxw7"]
Oct 30 19:21:48.675648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.675614    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"]
Oct 30 19:21:48.676412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676392    5065 cpu_manager.go:414] "containerMap: removing container" podUID="621023c3-d46d-48ca-9e22-dc7ce4505af0" containerName="revoke-keys"
Oct 30 19:21:48.676412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676413    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621023c3-d46d-48ca-9e22-dc7ce4505af0" containerName="revoke-keys"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676432    5065 cpu_manager.go:414] "containerMap: removing container" podUID="63b01866-a6da-4939-9bdb-83d970b07ef6" containerName="revoke-keys"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676438    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b01866-a6da-4939-9bdb-83d970b07ef6" containerName="revoke-keys"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676450    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676455    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676479    5065 cpu_manager.go:414] "containerMap: removing container" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676484    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676492    5065 cpu_manager.go:414] "containerMap: removing container" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676497    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676503    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676508    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676519    5065 cpu_manager.go:414] "containerMap: removing container" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing"
Oct 30 19:21:48.676554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676523    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676617    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="565da0d2-ac21-468f-aee9-af8467737307" containerName="payload-pre-processing"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676626    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="556e9d5c-0c3c-4ae8-8c7f-435d9a95120a" containerName="main"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676634    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f78b43-62f9-4fee-851a-8bf6d8cf70e5" containerName="main"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676640    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bee4cb-9318-42b1-b336-f664037b57e2" containerName="payload-processing"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676650    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="621023c3-d46d-48ca-9e22-dc7ce4505af0" containerName="revoke-keys"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676657    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b01866-a6da-4939-9bdb-83d970b07ef6" containerName="revoke-keys"
Oct 30 19:21:48.676926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.676662    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e44f70-dd7c-412a-be3d-6a2260040484" containerName="main"
Oct 30 19:21:48.681862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.681838    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.693044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.693002    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"]
Oct 30 19:21:48.755100 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.755059    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:48.755260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.755234    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:48.867665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.867633    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.867842 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.867770    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.867917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.867851    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbs4\" (UniqueName: \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.969411 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.969332    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbs4\" (UniqueName: \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.969902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.969845    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.970029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.969983    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.970515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.970492    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.972494 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.972476    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:48.988338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:48.988309    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbs4\" (UniqueName: \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:49.000117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:49.000077    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:49.321174 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:49.321142    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c357172_d031_4ca4_ba5c_f143b28b6515.slice/crio-b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031 WatchSource:0}: Error finding container b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031: Status 404 returned error can't find the container with id b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031
Oct 30 19:21:49.321539 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:49.321519    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"]
Oct 30 19:21:49.892493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:49.892461    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44b913c-e101-4e3c-bda5-cc3a2471c014" path="/var/lib/kubelet/pods/a44b913c-e101-4e3c-bda5-cc3a2471c014/volumes"
Oct 30 19:21:50.350405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:50.350360    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k" event={"ID":"1c357172-d031-4ca4-ba5c-f143b28b6515","Type":"ContainerStarted","Data":"61a3f1dcfdb5f76f5480da20beeda7b38dd367aa1bf6e0d5f22ee7958bb2757d"}
Oct 30 19:21:50.350876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:50.350435    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k" event={"ID":"1c357172-d031-4ca4-ba5c-f143b28b6515","Type":"ContainerStarted","Data":"b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031"}
Oct 30 19:21:50.446136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:50.444327    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k" podStartSLOduration=2.444306912 podStartE2EDuration="2.444306912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:50.44213171 +0000 UTC m=+2930.705596680" watchObservedRunningTime="2026-10-30 19:21:50.444306912 +0000 UTC m=+2930.707771863"
Oct 30 19:21:50.828955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:50.828918    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 19:21:50.829192 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:50.829168    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" containerID="cri-o://4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de" gracePeriod=30
Oct 30 19:21:51.361178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.361145    5065 generic.go:356] "Generic (PLEG): container finished" podID="1c357172-d031-4ca4-ba5c-f143b28b6515" containerID="61a3f1dcfdb5f76f5480da20beeda7b38dd367aa1bf6e0d5f22ee7958bb2757d" exitCode=0
Oct 30 19:21:51.361625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.361238    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k" event={"ID":"1c357172-d031-4ca4-ba5c-f143b28b6515","Type":"ContainerDied","Data":"61a3f1dcfdb5f76f5480da20beeda7b38dd367aa1bf6e0d5f22ee7958bb2757d"}
Oct 30 19:21:51.709597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.709503    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"]
Oct 30 19:21:51.726482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.726446    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da70a5-zlr6n"]
Oct 30 19:21:51.891533 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.891498    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e800cee6-7d63-49e8-95de-a4825e15e142" path="/var/lib/kubelet/pods/e800cee6-7d63-49e8-95de-a4825e15e142/volumes"
Oct 30 19:21:51.947283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.947250    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:51.948552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:51.948511    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:52.547413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.547387    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:52.623483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.623408    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\") pod \"1c357172-d031-4ca4-ba5c-f143b28b6515\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") "
Oct 30 19:21:52.623483 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.623469    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\") pod \"1c357172-d031-4ca4-ba5c-f143b28b6515\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") "
Oct 30 19:21:52.623723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.623539    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\" (UniqueName: \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\") pod \"1c357172-d031-4ca4-ba5c-f143b28b6515\" (UID: \"1c357172-d031-4ca4-ba5c-f143b28b6515\") "
Oct 30 19:21:52.624079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.624046    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca" pod "1c357172-d031-4ca4-ba5c-f143b28b6515" (UID: "1c357172-d031-4ca4-ba5c-f143b28b6515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:52.626845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.626817    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca" pod "1c357172-d031-4ca4-ba5c-f143b28b6515" (UID: "1c357172-d031-4ca4-ba5c-f143b28b6515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:52.627360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.627314    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4" pod "1c357172-d031-4ca4-ba5c-f143b28b6515" (UID: "1c357172-d031-4ca4-ba5c-f143b28b6515"). InnerVolumeSpecName "kube-api-access-bgbs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:52.725030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.725001    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:52.725030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.725031    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c357172-d031-4ca4-ba5c-f143b28b6515-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:52.725030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:52.725042    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgbs4\" (UniqueName: \"kubernetes.io/projected/1c357172-d031-4ca4-ba5c-f143b28b6515-kube-api-access-bgbs4\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:53.058234 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:53.058195    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:21:53.380636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:53.380558    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"
Oct 30 19:21:53.380777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:53.380562    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k" event={"ID":"1c357172-d031-4ca4-ba5c-f143b28b6515","Type":"ContainerDied","Data":"b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031"}
Oct 30 19:21:53.380777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:53.380685    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50032b4cfa0e0304e32abd2dc638c64eb766bfe41dfc15af49c15dde28ed031"
Oct 30 19:21:55.729601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:55.729569    5065 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:21:55.730018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:55.729624    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:21:56.199607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.199308    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 19:21:56.276382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276340    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276422    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276490    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276525    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276597    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276725    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.276795 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276741    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276905 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276855    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.276957 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276899    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.277007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.276965    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.277059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.277034    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\") pod \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\" (UID: \"e7956b75-cb3b-4443-9c51-c3c8d9dd2167\") "
Oct 30 19:21:56.277742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.277570    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.277742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.277590    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.278031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.278007    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:56.278387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.278366    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.278831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.278810    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.279734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.279708    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.282409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.282381    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:56.283030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.283007    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "kube-api-access-w8hb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.288216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.288191    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy" pod "e7956b75-cb3b-4443-9c51-c3c8d9dd2167" (UID: "e7956b75-cb3b-4443-9c51-c3c8d9dd2167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.379360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379282    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379322    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8hb9\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-kube-api-access-w8hb9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379341    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379355    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379370    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379385    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.379617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.379398    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7956b75-cb3b-4443-9c51-c3c8d9dd2167-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.419156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.419121    5065 generic.go:356] "Generic (PLEG): container finished" podID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerID="4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de" exitCode=0
Oct 30 19:21:56.419359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.419192    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"
Oct 30 19:21:56.419359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.419208    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerDied","Data":"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"}
Oct 30 19:21:56.419359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.419275    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2" event={"ID":"e7956b75-cb3b-4443-9c51-c3c8d9dd2167","Type":"ContainerDied","Data":"4642ec85d3712dd3833e6ee93dd05cfd4df04f6563c8bc693e6e0b9be396e81a"}
Oct 30 19:21:56.419359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.419305    5065 scope.go:122] "RemoveContainer" containerID="4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"
Oct 30 19:21:56.433748 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.433732    5065 scope.go:122] "RemoveContainer" containerID="c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"
Oct 30 19:21:56.444641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.444624    5065 scope.go:122] "RemoveContainer" containerID="4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"
Oct 30 19:21:56.444916 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:56.444894    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de\": container with ID starting with 4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de not found: ID does not exist" containerID="4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"
Oct 30 19:21:56.445005 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.444926    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de"} err="failed to get container status \"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de\": rpc error: code = NotFound desc = could not find container \"4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de\": container with ID starting with 4eaf64d5a246fbafe55618cae949a8a10a3071ac94bbc556772f1e71601da9de not found: ID does not exist"
Oct 30 19:21:56.445005 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.444943    5065 scope.go:122] "RemoveContainer" containerID="c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"
Oct 30 19:21:56.445202 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:56.445185    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f\": container with ID starting with c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f not found: ID does not exist" containerID="c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"
Oct 30 19:21:56.445251 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.445209    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f"} err="failed to get container status \"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f\": rpc error: code = NotFound desc = could not find container \"c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f\": container with ID starting with c4e040f28781e9831755ba97d37616320595ca5eafb85d4a39520e6a1201cd6f not found: ID does not exist"
Oct 30 19:21:56.453658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.453636    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 19:21:56.461793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.461753    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-bdcl2"]
Oct 30 19:21:56.511528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.511495    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 19:21:56.511778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:56.511753    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy" containerID="cri-o://a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c" gracePeriod=30
Oct 30 19:21:57.239731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.239692    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"]
Oct 30 19:21:57.240403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240387    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:21:57.240468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240405    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:21:57.240468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240444    5065 cpu_manager.go:414] "containerMap: removing container" podUID="1c357172-d031-4ca4-ba5c-f143b28b6515" containerName="revoke-keys"
Oct 30 19:21:57.240468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240449    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c357172-d031-4ca4-ba5c-f143b28b6515" containerName="revoke-keys"
Oct 30 19:21:57.240572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240551    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:21:57.240572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240560    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:21:57.240572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.240572    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c357172-d031-4ca4-ba5c-f143b28b6515" containerName="revoke-keys"
Oct 30 19:21:57.245798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.245780    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.256740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.256527    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"]
Oct 30 19:21:57.307059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.307018    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.307314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.307207    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.307830 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.307377    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.324731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.324699    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:57.324857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.324754    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:21:57.408855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.408826    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.409022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.408945    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.409163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.409147    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.409708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.409681    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.411790 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.411757    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.429758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.429734    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\") pod \"maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.564621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.564591    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:21:57.861517 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:21:57.861475    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815fe09e_d5b0_4f97_9881_9670d09eedd5.slice/crio-f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97 WatchSource:0}: Error finding container f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97: Status 404 returned error can't find the container with id f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97
Oct 30 19:21:57.862740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.862711    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"]
Oct 30 19:21:57.904137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:57.904068    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" path="/var/lib/kubelet/pods/e7956b75-cb3b-4443-9c51-c3c8d9dd2167/volumes"
Oct 30 19:21:58.445977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.445795    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn" event={"ID":"815fe09e-d5b0-4f97-9881-9670d09eedd5","Type":"ContainerStarted","Data":"4244bbce1d870b12fc8b1fe9727dc6dd0e3ce8ba078127f4bf6b1a6dacf450c9"}
Oct 30 19:21:58.445977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.445893    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn" event={"ID":"815fe09e-d5b0-4f97-9881-9670d09eedd5","Type":"ContainerStarted","Data":"f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97"}
Oct 30 19:21:58.471503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.471455    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn" podStartSLOduration=1.471442188 podStartE2EDuration="1.471442188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:58.469442415 +0000 UTC m=+2938.732907364" watchObservedRunningTime="2026-10-30 19:21:58.471442188 +0000 UTC m=+2938.734907135"
Oct 30 19:21:58.785116 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.784654    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 19:21:58.793630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.793601    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 19:21:58.840570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.839528    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\" (UniqueName: \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\") pod \"08bc11f0-5045-4097-a098-c36c2fca7630\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") "
Oct 30 19:21:58.840570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.839680    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\") pod \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") "
Oct 30 19:21:58.840570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.839849    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\") pod \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\" (UID: \"e62818c2-bcc0-4b64-9af7-9be58e7a0895\") "
Oct 30 19:21:58.840570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.839902    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\") pod \"08bc11f0-5045-4097-a098-c36c2fca7630\" (UID: \"08bc11f0-5045-4097-a098-c36c2fca7630\") "
Oct 30 19:21:58.840986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.840954    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume" pod "08bc11f0-5045-4097-a098-c36c2fca7630" (UID: "08bc11f0-5045-4097-a098-c36c2fca7630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:58.845270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.845237    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw" pod "08bc11f0-5045-4097-a098-c36c2fca7630" (UID: "08bc11f0-5045-4097-a098-c36c2fca7630"). InnerVolumeSpecName "kube-api-access-4qqtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:58.847352 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.847322    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume" pod "e62818c2-bcc0-4b64-9af7-9be58e7a0895" (UID: "e62818c2-bcc0-4b64-9af7-9be58e7a0895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:58.848870 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.848842    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx" pod "e62818c2-bcc0-4b64-9af7-9be58e7a0895" (UID: "e62818c2-bcc0-4b64-9af7-9be58e7a0895"). InnerVolumeSpecName "kube-api-access-22tjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:58.942695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.942665    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqtw\" (UniqueName: \"kubernetes.io/projected/08bc11f0-5045-4097-a098-c36c2fca7630-kube-api-access-4qqtw\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.942695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.942693    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tjx\" (UniqueName: \"kubernetes.io/projected/e62818c2-bcc0-4b64-9af7-9be58e7a0895-kube-api-access-22tjx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.942695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.942702    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62818c2-bcc0-4b64-9af7-9be58e7a0895-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:58.942695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:58.942712    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc11f0-5045-4097-a098-c36c2fca7630-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:59.458883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.458847    5065 generic.go:356] "Generic (PLEG): container finished" podID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerID="543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06" exitCode=0
Oct 30 19:21:59.459327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.458928    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"
Oct 30 19:21:59.459327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.458929    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" event={"ID":"e62818c2-bcc0-4b64-9af7-9be58e7a0895","Type":"ContainerDied","Data":"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"}
Oct 30 19:21:59.459327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.459027    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl" event={"ID":"e62818c2-bcc0-4b64-9af7-9be58e7a0895","Type":"ContainerDied","Data":"1fd4ccc8bb9626dae67853ce2bce86877ceb95d3a9f7a86946fcfd195690d457"}
Oct 30 19:21:59.459327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.459054    5065 scope.go:122] "RemoveContainer" containerID="543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"
Oct 30 19:21:59.461150 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.461118    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn" event={"ID":"815fe09e-d5b0-4f97-9881-9670d09eedd5","Type":"ContainerDied","Data":"4244bbce1d870b12fc8b1fe9727dc6dd0e3ce8ba078127f4bf6b1a6dacf450c9"}
Oct 30 19:21:59.461272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.461163    5065 generic.go:356] "Generic (PLEG): container finished" podID="815fe09e-d5b0-4f97-9881-9670d09eedd5" containerID="4244bbce1d870b12fc8b1fe9727dc6dd0e3ce8ba078127f4bf6b1a6dacf450c9" exitCode=0
Oct 30 19:21:59.462750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.462731    5065 generic.go:356] "Generic (PLEG): container finished" podID="08bc11f0-5045-4097-a098-c36c2fca7630" containerID="68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d" exitCode=0
Oct 30 19:21:59.462872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.462784    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"
Oct 30 19:21:59.462872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.462814    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" event={"ID":"08bc11f0-5045-4097-a098-c36c2fca7630","Type":"ContainerDied","Data":"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"}
Oct 30 19:21:59.462872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.462861    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t" event={"ID":"08bc11f0-5045-4097-a098-c36c2fca7630","Type":"ContainerDied","Data":"78a89c49bccc8c572292bc6b9fa5c1981d0d6f8927a52bbcfb42f41fe66491ae"}
Oct 30 19:21:59.471982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.471960    5065 scope.go:122] "RemoveContainer" containerID="543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"
Oct 30 19:21:59.472323 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:59.472304    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06\": container with ID starting with 543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06 not found: ID does not exist" containerID="543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"
Oct 30 19:21:59.472417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.472335    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06"} err="failed to get container status \"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06\": rpc error: code = NotFound desc = could not find container \"543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06\": container with ID starting with 543aa1efdf1b4dccad85d297ecc27cf47e40471d3fbdf6b83b7ec369a7f89d06 not found: ID does not exist"
Oct 30 19:21:59.472417 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.472356    5065 scope.go:122] "RemoveContainer" containerID="68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"
Oct 30 19:21:59.484622 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.484603    5065 scope.go:122] "RemoveContainer" containerID="68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"
Oct 30 19:21:59.484923 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:21:59.484907    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d\": container with ID starting with 68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d not found: ID does not exist" containerID="68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"
Oct 30 19:21:59.484986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.484929    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d"} err="failed to get container status \"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d\": rpc error: code = NotFound desc = could not find container \"68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d\": container with ID starting with 68fff280b89f0d96b485d9759e52d55e97e311eaa4a63b8dc561e161791d223d not found: ID does not exist"
Oct 30 19:21:59.591118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.588901    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 19:21:59.615666 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.615636    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da70a5-8f867f6bc-bv2pl"]
Oct 30 19:21:59.660980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.660953    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 19:21:59.674663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.674636    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da70a5-777bb79b78-7n25t"]
Oct 30 19:21:59.965207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.965143    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" path="/var/lib/kubelet/pods/08bc11f0-5045-4097-a098-c36c2fca7630/volumes"
Oct 30 19:21:59.965782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.965729    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" path="/var/lib/kubelet/pods/e62818c2-bcc0-4b64-9af7-9be58e7a0895/volumes"
Oct 30 19:21:59.966172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.966154    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 19:21:59.966386 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:21:59.966363    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy" containerID="cri-o://13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7" gracePeriod=30
Oct 30 19:22:00.670275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.670252    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:22:00.828492 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.828452    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\") pod \"815fe09e-d5b0-4f97-9881-9670d09eedd5\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") "
Oct 30 19:22:00.828706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.828555    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\") pod \"815fe09e-d5b0-4f97-9881-9670d09eedd5\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") "
Oct 30 19:22:00.828706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.828613    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\") pod \"815fe09e-d5b0-4f97-9881-9670d09eedd5\" (UID: \"815fe09e-d5b0-4f97-9881-9670d09eedd5\") "
Oct 30 19:22:00.828900 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.828798    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca" pod "815fe09e-d5b0-4f97-9881-9670d09eedd5" (UID: "815fe09e-d5b0-4f97-9881-9670d09eedd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:00.829053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.829028    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:00.831042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.831010    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca" pod "815fe09e-d5b0-4f97-9881-9670d09eedd5" (UID: "815fe09e-d5b0-4f97-9881-9670d09eedd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:00.831507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.831436    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz" pod "815fe09e-d5b0-4f97-9881-9670d09eedd5" (UID: "815fe09e-d5b0-4f97-9881-9670d09eedd5"). InnerVolumeSpecName "kube-api-access-2xkdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:00.930856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.930819    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/815fe09e-d5b0-4f97-9881-9670d09eedd5-kube-api-access-2xkdz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:00.930856 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:00.930853    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815fe09e-d5b0-4f97-9881-9670d09eedd5-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:01.510601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:01.510572    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"
Oct 30 19:22:01.510810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:01.510606    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn" event={"ID":"815fe09e-d5b0-4f97-9881-9670d09eedd5","Type":"ContainerDied","Data":"f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97"}
Oct 30 19:22:01.510810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:01.510659    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a904ce159aa8779b9441050015d5751dc16f1a471bdb6b7a2db2cebabebf97"
Oct 30 19:22:02.079284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.076400    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 19:22:02.122140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.116973    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 19:22:02.132714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.132685    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:22:02.149742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.149719    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.149855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.149792    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.149907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.149879    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.149947 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.149933    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150006    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150102 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150064    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150126    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150203    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150419 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150320    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\") pod \"229795d4-f03f-4940-a980-59606344d520\" (UID: \"229795d4-f03f-4940-a980-59606344d520\") "
Oct 30 19:22:02.150518 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150423    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:02.151165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.150889    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.151661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.151307    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:02.151661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.151432    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:02.151661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.151438    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:02.151911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.151731    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:02.152886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.152860    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:02.153673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.153652    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:02.153879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.153858    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:02.154791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.154756    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt" pod "229795d4-f03f-4940-a980-59606344d520" (UID: "229795d4-f03f-4940-a980-59606344d520"). InnerVolumeSpecName "kube-api-access-8mnxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:02.252266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252229    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252260    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252270    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252279    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252289    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/229795d4-f03f-4940-a980-59606344d520-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252298    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/229795d4-f03f-4940-a980-59606344d520-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252306    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mnxt\" (UniqueName: \"kubernetes.io/projected/229795d4-f03f-4940-a980-59606344d520-kube-api-access-8mnxt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.252546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.252328    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/229795d4-f03f-4940-a980-59606344d520-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.524161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.524121    5065 generic.go:356] "Generic (PLEG): container finished" podID="229795d4-f03f-4940-a980-59606344d520" containerID="a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c" exitCode=0
Oct 30 19:22:02.524788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.524362    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerDied","Data":"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"}
Oct 30 19:22:02.524788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.524417    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m" event={"ID":"229795d4-f03f-4940-a980-59606344d520","Type":"ContainerDied","Data":"e0645603b888f90fcf89859cf2ba3ea73790d5800f3e7e6f1907b40477fac3db"}
Oct 30 19:22:02.524788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.524446    5065 scope.go:122] "RemoveContainer" containerID="a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"
Oct 30 19:22:02.524788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.524603    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"
Oct 30 19:22:02.608673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.608652    5065 scope.go:122] "RemoveContainer" containerID="a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"
Oct 30 19:22:02.621426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.621392    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 19:22:02.621688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.621671    5065 scope.go:122] "RemoveContainer" containerID="a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"
Oct 30 19:22:02.622021 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:02.621995    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c\": container with ID starting with a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c not found: ID does not exist" containerID="a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"
Oct 30 19:22:02.622189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.622033    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c"} err="failed to get container status \"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c\": rpc error: code = NotFound desc = could not find container \"a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c\": container with ID starting with a72304d09dcd60002f81f94ccdef36e0f8d82530884bce08cf243c7f5cfde96c not found: ID does not exist"
Oct 30 19:22:02.622189 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.622059    5065 scope.go:122] "RemoveContainer" containerID="a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"
Oct 30 19:22:02.622432 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:02.622409    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520\": container with ID starting with a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520 not found: ID does not exist" containerID="a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"
Oct 30 19:22:02.622506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.622443    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520"} err="failed to get container status \"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520\": rpc error: code = NotFound desc = could not find container \"a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520\": container with ID starting with a8efb13c737661581db3c3426e12d46fc54b9f1c7325432a25b9f854b8400520 not found: ID does not exist"
Oct 30 19:22:02.627221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:02.627195    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-gw74m"]
Oct 30 19:22:03.894323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:03.894286    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229795d4-f03f-4940-a980-59606344d520" path="/var/lib/kubelet/pods/229795d4-f03f-4940-a980-59606344d520/volumes"
Oct 30 19:22:05.308347 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.308321    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 19:22:05.407767 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.407730    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.407814    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.407866    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.407912    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.407990    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408157    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408195    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408282    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.408387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408368    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\") pod \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\" (UID: \"f163f3ff-6512-4081-8e1b-17f8bb6ddf55\") "
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408194    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.408960    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409198    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409387    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409693    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409717    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409731    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.409745    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.410875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.410819    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:05.416377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.416340    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "kube-api-access-px574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:05.419297 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.419253    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:05.420802 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.420237    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:05.421026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.420994    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo" pod "f163f3ff-6512-4081-8e1b-17f8bb6ddf55" (UID: "f163f3ff-6512-4081-8e1b-17f8bb6ddf55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:05.511455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.511429    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.511455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.511453    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.511662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.511463    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.511662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.511472    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px574\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-kube-api-access-px574\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.511662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.511481    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f163f3ff-6512-4081-8e1b-17f8bb6ddf55-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.554012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.553947    5065 generic.go:356] "Generic (PLEG): container finished" podID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerID="13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7" exitCode=0
Oct 30 19:22:05.554153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.554007    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"
Oct 30 19:22:05.554153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.554023    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" event={"ID":"f163f3ff-6512-4081-8e1b-17f8bb6ddf55","Type":"ContainerDied","Data":"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"}
Oct 30 19:22:05.554153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.554077    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6" event={"ID":"f163f3ff-6512-4081-8e1b-17f8bb6ddf55","Type":"ContainerDied","Data":"1db0ba5f6dab37ad6c7d40c0e9404482b4764ee71901b7fdb90df25230e526d2"}
Oct 30 19:22:05.554320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.554139    5065 scope.go:122] "RemoveContainer" containerID="13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"
Oct 30 19:22:05.573550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.573507    5065 scope.go:122] "RemoveContainer" containerID="13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"
Oct 30 19:22:05.573950 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:05.573918    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7\": container with ID starting with 13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7 not found: ID does not exist" containerID="13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"
Oct 30 19:22:05.574142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.574114    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7"} err="failed to get container status \"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7\": rpc error: code = NotFound desc = could not find container \"13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7\": container with ID starting with 13263d0f54bd681b6cc3c8bfcdffc6ea959c305211be6aefd5b99501b97832c7 not found: ID does not exist"
Oct 30 19:22:05.591500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.591473    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 19:22:05.599201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.599179    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pjhv6"]
Oct 30 19:22:05.891690 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:05.891612    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" path="/var/lib/kubelet/pods/f163f3ff-6512-4081-8e1b-17f8bb6ddf55/volumes"
Oct 30 19:22:06.581943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.581903    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"]
Oct 30 19:22:06.582552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582536    5065 cpu_manager.go:414] "containerMap: removing container" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy"
Oct 30 19:22:06.582552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582552    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582564    5065 cpu_manager.go:414] "containerMap: removing container" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582569    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582583    5065 cpu_manager.go:414] "containerMap: removing container" podUID="815fe09e-d5b0-4f97-9881-9670d09eedd5" containerName="revoke-keys"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582589    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815fe09e-d5b0-4f97-9881-9670d09eedd5" containerName="revoke-keys"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582597    5065 cpu_manager.go:414] "containerMap: removing container" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582601    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582626    5065 cpu_manager.go:414] "containerMap: removing container" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582630    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582644    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582648    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7956b75-cb3b-4443-9c51-c3c8d9dd2167" containerName="istio-proxy"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582659    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing"
Oct 30 19:22:06.582668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582664    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing"
Oct 30 19:22:06.583404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582764    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62818c2-bcc0-4b64-9af7-9be58e7a0895" containerName="payload-pre-processing"
Oct 30 19:22:06.583404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582775    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="f163f3ff-6512-4081-8e1b-17f8bb6ddf55" containerName="istio-proxy"
Oct 30 19:22:06.583404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582783    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:06.583404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582789    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="815fe09e-d5b0-4f97-9881-9670d09eedd5" containerName="revoke-keys"
Oct 30 19:22:06.583404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.582797    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bc11f0-5045-4097-a098-c36c2fca7630" containerName="payload-processing"
Oct 30 19:22:06.587920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.587897    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.595196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.595178    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:06.595419 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.595384    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:22:06.597284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.597069    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"]
Oct 30 19:22:06.630755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.630730    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.630886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.630789    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.630967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.630942    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrf2\" (UniqueName: \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.731976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.731941    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.732153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.731999    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.732153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.732050    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrf2\" (UniqueName: \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.732710 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.732639    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.734535 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.734512    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.751215 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.751191    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrf2\" (UniqueName: \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\") pod \"maas-api-revoke-sub-11e9bdd6-vn8jf\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:06.918854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:06.918785    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:07.222315 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:07.222285    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"]
Oct 30 19:22:07.223192 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:07.223166    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b37d90f_b5af_4ebd_a774_3f57e0bdaa4c.slice/crio-826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14 WatchSource:0}: Error finding container 826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14: Status 404 returned error can't find the container with id 826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14
Oct 30 19:22:07.504937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:07.504480    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 19:22:07.574457 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:07.574419    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf" event={"ID":"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c","Type":"ContainerStarted","Data":"826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14"}
Oct 30 19:22:08.584398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:08.584311    5065 generic.go:356] "Generic (PLEG): container finished" podID="2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" containerID="937da0ba203fb3f3b1f523c6e684eb17a2256941d98b053eb385202669170503" exitCode=0
Oct 30 19:22:08.584786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:08.584393    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf" event={"ID":"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c","Type":"ContainerDied","Data":"937da0ba203fb3f3b1f523c6e684eb17a2256941d98b053eb385202669170503"}
Oct 30 19:22:09.031122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.031063    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"]
Oct 30 19:22:09.035939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.035916    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673606ee-kxrpd"]
Oct 30 19:22:09.768726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.768704    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:09.886961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.886878    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\" (UniqueName: \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\") pod \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") "
Oct 30 19:22:09.886961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.886943    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\") pod \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") "
Oct 30 19:22:09.887231 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.886978    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\") pod \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\" (UID: \"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c\") "
Oct 30 19:22:09.887711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.887678    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca" pod "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" (UID: "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:09.891530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.891492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2" pod "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" (UID: "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c"). InnerVolumeSpecName "kube-api-access-wcrf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:09.894456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.894428    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e86642d-7fc4-4171-bb7f-0221ea30334e" path="/var/lib/kubelet/pods/2e86642d-7fc4-4171-bb7f-0221ea30334e/volumes"
Oct 30 19:22:09.895634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.895599    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca" pod "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" (UID: "2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:09.990602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.990556    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrf2\" (UniqueName: \"kubernetes.io/projected/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-kube-api-access-wcrf2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:09.990878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.990861    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:09.991029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:09.991015    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:10.170801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.170729    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"]
Oct 30 19:22:10.171391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.171377    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" containerName="revoke-keys"
Oct 30 19:22:10.171449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.171393    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" containerName="revoke-keys"
Oct 30 19:22:10.171511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.171502    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="229795d4-f03f-4940-a980-59606344d520" containerName="istio-proxy"
Oct 30 19:22:10.171551 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.171515    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" containerName="revoke-keys"
Oct 30 19:22:10.176479 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.176454    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.232124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.231424    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"]
Oct 30 19:22:10.294110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.287277    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"]
Oct 30 19:22:10.294110 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.287406    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.379468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.379422    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.379680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.379556    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.379782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.379763    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.424403 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.424292    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"]
Oct 30 19:22:10.459378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.459314    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"]
Oct 30 19:22:10.465649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.465621    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-21d896-6qc5k"]
Oct 30 19:22:10.481693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.481662    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.481857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.481781    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.481857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.481818    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.482669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.481996    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.482669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.482031    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.482669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.482071    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.482953 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.482804    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.485523 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.485490    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.498742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.498717    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\") pod \"maas-api-revoke-sub-02c1d4c1-c58nt\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.578421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.578399    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:10.583503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.583475    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.583692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.583673    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.583772 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.583721    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.584235 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.584209    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.586402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.586375    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.647409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.647337    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\") pod \"maas-api-revoke-sub-114e9063-dv42l\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.661224 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.661191    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"
Oct 30 19:22:10.661391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.661239    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf" event={"ID":"2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c","Type":"ContainerDied","Data":"826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14"}
Oct 30 19:22:10.661391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.661294    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="826eae2c839035c98df5a000169021ea53be5fe466e9c5b1fa6180324578cd14"
Oct 30 19:22:10.684594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.684529    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:10.942481 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:10.942453    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b5c8ab_6c70_417d_b865_7384417f7041.slice/crio-de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a WatchSource:0}: Error finding container de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a: Status 404 returned error can't find the container with id de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a
Oct 30 19:22:10.943140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.943117    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"]
Oct 30 19:22:10.997661 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:10.997634    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"]
Oct 30 19:22:11.001485 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:11.001454    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d5e362_913a_49d5_a4cb_981c7a289443.slice/crio-b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902 WatchSource:0}: Error finding container b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902: Status 404 returned error can't find the container with id b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902
Oct 30 19:22:11.708042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.708000    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt" event={"ID":"08b5c8ab-6c70-417d-b865-7384417f7041","Type":"ContainerStarted","Data":"613adb1a695ceebbf8f2a6302e4ed5d99c7c2859ced488088a722ffd0afd3d18"}
Oct 30 19:22:11.708289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.708069    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt" event={"ID":"08b5c8ab-6c70-417d-b865-7384417f7041","Type":"ContainerStarted","Data":"de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a"}
Oct 30 19:22:11.711843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.711792    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l" event={"ID":"c6d5e362-913a-49d5-a4cb-981c7a289443","Type":"ContainerStarted","Data":"3d5d70f687b4a4a9a5b609aba410141b8da1d2c02ed4a2affee95f3816136199"}
Oct 30 19:22:11.711966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.711872    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l" event={"ID":"c6d5e362-913a-49d5-a4cb-981c7a289443","Type":"ContainerStarted","Data":"b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902"}
Oct 30 19:22:11.729996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.729944    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt" podStartSLOduration=1.729929837 podStartE2EDuration="1.729929837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:11.728967746 +0000 UTC m=+2951.992432719" watchObservedRunningTime="2026-10-30 19:22:11.729929837 +0000 UTC m=+2951.993394785"
Oct 30 19:22:11.752881 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.752841    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l" podStartSLOduration=1.752829368 podStartE2EDuration="1.752829368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:11.750507076 +0000 UTC m=+2952.013972045" watchObservedRunningTime="2026-10-30 19:22:11.752829368 +0000 UTC m=+2952.016294315"
Oct 30 19:22:11.893607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:11.893563    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c357172-d031-4ca4-ba5c-f143b28b6515" path="/var/lib/kubelet/pods/1c357172-d031-4ca4-ba5c-f143b28b6515/volumes"
Oct 30 19:22:12.506365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:12.506160    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing" containerID="cri-o://d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e" gracePeriod=25
Oct 30 19:22:12.728331 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:12.728291    5065 generic.go:356] "Generic (PLEG): container finished" podID="c6d5e362-913a-49d5-a4cb-981c7a289443" containerID="3d5d70f687b4a4a9a5b609aba410141b8da1d2c02ed4a2affee95f3816136199" exitCode=0
Oct 30 19:22:12.728899 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:12.728869    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l" event={"ID":"c6d5e362-913a-49d5-a4cb-981c7a289443","Type":"ContainerDied","Data":"3d5d70f687b4a4a9a5b609aba410141b8da1d2c02ed4a2affee95f3816136199"}
Oct 30 19:22:12.731726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:12.731695    5065 generic.go:356] "Generic (PLEG): container finished" podID="08b5c8ab-6c70-417d-b865-7384417f7041" containerID="613adb1a695ceebbf8f2a6302e4ed5d99c7c2859ced488088a722ffd0afd3d18" exitCode=0
Oct 30 19:22:12.731839 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:12.731779    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt" event={"ID":"08b5c8ab-6c70-417d-b865-7384417f7041","Type":"ContainerDied","Data":"613adb1a695ceebbf8f2a6302e4ed5d99c7c2859ced488088a722ffd0afd3d18"}
Oct 30 19:22:13.949270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:13.949239    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:13.981945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:13.981920    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:14.038425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.038382    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\") pod \"08b5c8ab-6c70-417d-b865-7384417f7041\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") "
Oct 30 19:22:14.039416 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.038734    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\") pod \"c6d5e362-913a-49d5-a4cb-981c7a289443\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") "
Oct 30 19:22:14.041996 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.041973    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\") pod \"08b5c8ab-6c70-417d-b865-7384417f7041\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") "
Oct 30 19:22:14.042131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.042047    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\") pod \"c6d5e362-913a-49d5-a4cb-981c7a289443\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") "
Oct 30 19:22:14.042459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.042439    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\") pod \"08b5c8ab-6c70-417d-b865-7384417f7041\" (UID: \"08b5c8ab-6c70-417d-b865-7384417f7041\") "
Oct 30 19:22:14.042817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.042790    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca" pod "08b5c8ab-6c70-417d-b865-7384417f7041" (UID: "08b5c8ab-6c70-417d-b865-7384417f7041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:14.042943 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.042924    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\") pod \"c6d5e362-913a-49d5-a4cb-981c7a289443\" (UID: \"c6d5e362-913a-49d5-a4cb-981c7a289443\") "
Oct 30 19:22:14.043384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.043342    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca" pod "c6d5e362-913a-49d5-a4cb-981c7a289443" (UID: "c6d5e362-913a-49d5-a4cb-981c7a289443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:14.044283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.044237    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.044283 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.044265    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.044732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.044689    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g" pod "08b5c8ab-6c70-417d-b865-7384417f7041" (UID: "08b5c8ab-6c70-417d-b865-7384417f7041"). InnerVolumeSpecName "kube-api-access-qbk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:14.045070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.045048    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca" pod "c6d5e362-913a-49d5-a4cb-981c7a289443" (UID: "c6d5e362-913a-49d5-a4cb-981c7a289443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:14.045548 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.045497    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca" pod "08b5c8ab-6c70-417d-b865-7384417f7041" (UID: "08b5c8ab-6c70-417d-b865-7384417f7041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:14.046801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.046776    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8" pod "c6d5e362-913a-49d5-a4cb-981c7a289443" (UID: "c6d5e362-913a-49d5-a4cb-981c7a289443"). InnerVolumeSpecName "kube-api-access-vk7h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:14.145424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.145397    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b5c8ab-6c70-417d-b865-7384417f7041-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.145424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.145425    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/c6d5e362-913a-49d5-a4cb-981c7a289443-kube-api-access-vk7h8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.145595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.145436    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/08b5c8ab-6c70-417d-b865-7384417f7041-kube-api-access-qbk5g\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.145595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.145446    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d5e362-913a-49d5-a4cb-981c7a289443-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.749425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.749399    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"
Oct 30 19:22:14.749603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.749440    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l" event={"ID":"c6d5e362-913a-49d5-a4cb-981c7a289443","Type":"ContainerDied","Data":"b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902"}
Oct 30 19:22:14.749603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.749490    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0680a4ee57898d994215d632b2207d5abe089567438d4f59bf2e03bd6b43902"
Oct 30 19:22:14.750992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.750961    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"
Oct 30 19:22:14.750992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.750970    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt" event={"ID":"08b5c8ab-6c70-417d-b865-7384417f7041","Type":"ContainerDied","Data":"de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a"}
Oct 30 19:22:14.751179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.751029    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de3aa978c19830b1fb58f3b512109e169e9ae4622b83f10a0a7addb253b4c08a"
Oct 30 19:22:14.984860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:14.984829    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:17.141185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.135261    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" containerID="cri-o://459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9" gracePeriod=45
Oct 30 19:22:17.157623 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.157414    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main" containerID="cri-o://acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c" gracePeriod=45
Oct 30 19:22:17.453817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.453787    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"]
Oct 30 19:22:17.454794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.454773    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c6d5e362-913a-49d5-a4cb-981c7a289443" containerName="revoke-keys"
Oct 30 19:22:17.454868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.454799    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d5e362-913a-49d5-a4cb-981c7a289443" containerName="revoke-keys"
Oct 30 19:22:17.454906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.454875    5065 cpu_manager.go:414] "containerMap: removing container" podUID="08b5c8ab-6c70-417d-b865-7384417f7041" containerName="revoke-keys"
Oct 30 19:22:17.454906 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.454883    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b5c8ab-6c70-417d-b865-7384417f7041" containerName="revoke-keys"
Oct 30 19:22:17.455057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.455043    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b5c8ab-6c70-417d-b865-7384417f7041" containerName="revoke-keys"
Oct 30 19:22:17.455119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.455071    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d5e362-913a-49d5-a4cb-981c7a289443" containerName="revoke-keys"
Oct 30 19:22:17.459595 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.459572    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.476357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.476333    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:17.476502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.476448    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:22:17.489141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.480067    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"]
Oct 30 19:22:17.500329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.500298    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:22:17.545850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.545808    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.546028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.545897    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st84m\" (UniqueName: \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.546028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.545979    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.563630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.563603    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:22:17.647464 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647428    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.647643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647507    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.647643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647551    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.647797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647647    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.647797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647700    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.647797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647755    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.647797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647778    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647795    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.648036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647881    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.648036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647926    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.648036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647945    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648036 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.647982    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.648342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648056    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\" (UniqueName: \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\") pod \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\" (UID: \"cfed93ae-e928-4a4d-9bb2-00a25f1989b6\") "
Oct 30 19:22:17.648342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648135    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\" (UniqueName: \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\") pod \"b4245d03-203c-4837-96d0-69e96e66b522\" (UID: \"b4245d03-203c-4837-96d0-69e96e66b522\") "
Oct 30 19:22:17.648342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648198    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648390    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648519 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648400    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.648802 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648577    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648802 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648611    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.648921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648845    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.648921 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.648913    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st84m\" (UniqueName: \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.649058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649036    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.649136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649064    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.649212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649155    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.649212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649168    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.649212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649182    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.649212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.649196    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.650254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.650226    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.651132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.651077    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.652340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.652304    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:17.652340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.652317    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "kube-api-access-nz87m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:17.652340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.652333    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm" pod "b4245d03-203c-4837-96d0-69e96e66b522" (UID: "b4245d03-203c-4837-96d0-69e96e66b522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.652783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.652754    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.653223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.653203    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:17.654187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.654160    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc" pod "cfed93ae-e928-4a4d-9bb2-00a25f1989b6" (UID: "cfed93ae-e928-4a4d-9bb2-00a25f1989b6"). InnerVolumeSpecName "kube-api-access-vr6hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:17.664262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.664233    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st84m\" (UniqueName: \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\") pod \"maas-api-revoke-sub-7fa00328-l4k7s\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750687    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6hc\" (UniqueName: \"kubernetes.io/projected/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-kube-api-access-vr6hc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750714    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz87m\" (UniqueName: \"kubernetes.io/projected/b4245d03-203c-4837-96d0-69e96e66b522-kube-api-access-nz87m\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750725    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4245d03-203c-4837-96d0-69e96e66b522-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750736    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750745    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfed93ae-e928-4a4d-9bb2-00a25f1989b6-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.750760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.750753    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4245d03-203c-4837-96d0-69e96e66b522-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.777426 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.777397    5065 generic.go:356] "Generic (PLEG): container finished" podID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerID="459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9" exitCode=0
Oct 30 19:22:17.777582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.777453    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"
Oct 30 19:22:17.777582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.777484    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" event={"ID":"cfed93ae-e928-4a4d-9bb2-00a25f1989b6","Type":"ContainerDied","Data":"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"}
Oct 30 19:22:17.777582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.777535    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" event={"ID":"cfed93ae-e928-4a4d-9bb2-00a25f1989b6","Type":"ContainerDied","Data":"52a5ad3459076e8f15f796e43b7521df5f2f2a2b94c739164e95f199d00d2988"}
Oct 30 19:22:17.777582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.777550    5065 scope.go:122] "RemoveContainer" containerID="459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"
Oct 30 19:22:17.779167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.779140    5065 generic.go:356] "Generic (PLEG): container finished" podID="b4245d03-203c-4837-96d0-69e96e66b522" containerID="acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c" exitCode=0
Oct 30 19:22:17.779383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.779327    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"
Oct 30 19:22:17.779383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.779340    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" event={"ID":"b4245d03-203c-4837-96d0-69e96e66b522","Type":"ContainerDied","Data":"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"}
Oct 30 19:22:17.779383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.779371    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f" event={"ID":"b4245d03-203c-4837-96d0-69e96e66b522","Type":"ContainerDied","Data":"7b4ecfa432306b5031478300f06d120d2c5f24e04b2028acc91284416f344d03"}
Oct 30 19:22:17.792812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.792797    5065 scope.go:122] "RemoveContainer" containerID="459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"
Oct 30 19:22:17.793058 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:17.793039    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9\": container with ID starting with 459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9 not found: ID does not exist" containerID="459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"
Oct 30 19:22:17.793230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.793068    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9"} err="failed to get container status \"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9\": rpc error: code = NotFound desc = could not find container \"459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9\": container with ID starting with 459098dbc5e87b02322ab79f6eddc543ffd4435dc8c9ebb7f0746688c4b0afc9 not found: ID does not exist"
Oct 30 19:22:17.793230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.793138    5065 scope.go:122] "RemoveContainer" containerID="acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"
Oct 30 19:22:17.805168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.805147    5065 scope.go:122] "RemoveContainer" containerID="acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"
Oct 30 19:22:17.805470 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:17.805451    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c\": container with ID starting with acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c not found: ID does not exist" containerID="acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"
Oct 30 19:22:17.805554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.805476    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c"} err="failed to get container status \"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c\": rpc error: code = NotFound desc = could not find container \"acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c\": container with ID starting with acb5c0116be4b6b68e82df512e07e28e469bf7a6b9ce4d987e701ef4daf0474c not found: ID does not exist"
Oct 30 19:22:17.814219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.814194    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 19:22:17.819216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.819195    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89"]
Oct 30 19:22:17.846053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.846023    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 19:22:17.850158 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.850135    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/simulator-w0-84c32a-kserve-765556f5c8-95w7f"]
Oct 30 19:22:17.862547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.862522    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:17.894930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.894905    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4245d03-203c-4837-96d0-69e96e66b522" path="/var/lib/kubelet/pods/b4245d03-203c-4837-96d0-69e96e66b522/volumes"
Oct 30 19:22:17.895373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:17.895357    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" path="/var/lib/kubelet/pods/cfed93ae-e928-4a4d-9bb2-00a25f1989b6/volumes"
Oct 30 19:22:18.168983 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:18.168934    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d4e518_56e0_4938_bc8e_63a380297efa.slice/crio-ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4 WatchSource:0}: Error finding container ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4: Status 404 returned error can't find the container with id ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4
Oct 30 19:22:18.171074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:18.171045    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"]
Oct 30 19:22:18.415998 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:18.415904    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/premium-w0-84c32a-kserve-bcccf7bc9-wzc89" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:18.793980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:18.793941    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s" event={"ID":"27d4e518-56e0-4938-bc8e-63a380297efa","Type":"ContainerStarted","Data":"adf8b2fa20d81d409fbbc35fd36426bef8cb7900129af3e644c5eb50ec10f346"}
Oct 30 19:22:18.794198 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:18.794018    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s" event={"ID":"27d4e518-56e0-4938-bc8e-63a380297efa","Type":"ContainerStarted","Data":"ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4"}
Oct 30 19:22:18.816324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:18.816279    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s" podStartSLOduration=1.816263331 podStartE2EDuration="1.816263331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:18.815806996 +0000 UTC m=+2959.079271946" watchObservedRunningTime="2026-10-30 19:22:18.816263331 +0000 UTC m=+2959.079728280"
Oct 30 19:22:19.249658 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.249622    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:22:19.250205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.249875    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy" containerID="cri-o://6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3" gracePeriod=30
Oct 30 19:22:19.706373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.706333    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"]
Oct 30 19:22:19.707333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707307    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main"
Oct 30 19:22:19.707333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707329    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main"
Oct 30 19:22:19.707499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707345    5065 cpu_manager.go:414] "containerMap: removing container" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main"
Oct 30 19:22:19.707499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707353    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main"
Oct 30 19:22:19.707570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707515    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4245d03-203c-4837-96d0-69e96e66b522" containerName="main"
Oct 30 19:22:19.707570 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.707528    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfed93ae-e928-4a4d-9bb2-00a25f1989b6" containerName="main"
Oct 30 19:22:19.711922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.711901    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.719616 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.719596    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"]
Oct 30 19:22:19.804577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.804542    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.804728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.804645    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.804808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.804740    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjq2t\" (UniqueName: \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.805184 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.805157    5065 generic.go:356] "Generic (PLEG): container finished" podID="27d4e518-56e0-4938-bc8e-63a380297efa" containerID="adf8b2fa20d81d409fbbc35fd36426bef8cb7900129af3e644c5eb50ec10f346" exitCode=0
Oct 30 19:22:19.805279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.805246    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s" event={"ID":"27d4e518-56e0-4938-bc8e-63a380297efa","Type":"ContainerDied","Data":"adf8b2fa20d81d409fbbc35fd36426bef8cb7900129af3e644c5eb50ec10f346"}
Oct 30 19:22:19.906073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.906038    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq2t\" (UniqueName: \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.906949 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.906921    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.907130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.906969    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.908177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.908152    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.914173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.910230    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:19.928820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:19.921337    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjq2t\" (UniqueName: \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\") pod \"maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:20.032738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:20.032714    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:20.328308 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:20.328272    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93da129d_b9e3_4256_aa8b_bcc6e50556f8.slice/crio-426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9 WatchSource:0}: Error finding container 426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9: Status 404 returned error can't find the container with id 426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9
Oct 30 19:22:20.328654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:20.328494    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"]
Oct 30 19:22:20.818186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:20.818140    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b" event={"ID":"93da129d-b9e3-4256-aa8b-bcc6e50556f8","Type":"ContainerStarted","Data":"6023e333939437ec74f67b6fc831c2525ab44e12de524c6f76c4f5ae8630a051"}
Oct 30 19:22:20.818355 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:20.818203    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b" event={"ID":"93da129d-b9e3-4256-aa8b-bcc6e50556f8","Type":"ContainerStarted","Data":"426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9"}
Oct 30 19:22:20.849972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:20.849868    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b" podStartSLOduration=1.849831937 podStartE2EDuration="1.849831937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:20.844499362 +0000 UTC m=+2961.107964306" watchObservedRunningTime="2026-10-30 19:22:20.849831937 +0000 UTC m=+2961.113296885"
Oct 30 19:22:21.021167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.021133    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:21.040515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.040477    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"]
Oct 30 19:22:21.046413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.046365    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703002e-fzpx6"]
Oct 30 19:22:21.066891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.066858    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\") pod \"27d4e518-56e0-4938-bc8e-63a380297efa\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") "
Oct 30 19:22:21.067056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.067023    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\") pod \"27d4e518-56e0-4938-bc8e-63a380297efa\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") "
Oct 30 19:22:21.067180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.067139    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\" (UniqueName: \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\") pod \"27d4e518-56e0-4938-bc8e-63a380297efa\" (UID: \"27d4e518-56e0-4938-bc8e-63a380297efa\") "
Oct 30 19:22:21.067314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.067289    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca" pod "27d4e518-56e0-4938-bc8e-63a380297efa" (UID: "27d4e518-56e0-4938-bc8e-63a380297efa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:21.067831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.067770    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:21.070346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.070314    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca" pod "27d4e518-56e0-4938-bc8e-63a380297efa" (UID: "27d4e518-56e0-4938-bc8e-63a380297efa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:21.070454 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.070361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m" pod "27d4e518-56e0-4938-bc8e-63a380297efa" (UID: "27d4e518-56e0-4938-bc8e-63a380297efa"). InnerVolumeSpecName "kube-api-access-st84m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:21.169128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.169019    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d4e518-56e0-4938-bc8e-63a380297efa-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:21.169128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.169059    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st84m\" (UniqueName: \"kubernetes.io/projected/27d4e518-56e0-4938-bc8e-63a380297efa-kube-api-access-st84m\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:21.833402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.831325    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s" event={"ID":"27d4e518-56e0-4938-bc8e-63a380297efa","Type":"ContainerDied","Data":"ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4"}
Oct 30 19:22:21.833402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.831389    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca96437bc0a158911d114f319edc0288a69b6c1f36552f0f35d01a1c15e4a2f4"
Oct 30 19:22:21.833402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.831459    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"
Oct 30 19:22:21.838769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.838737    5065 generic.go:356] "Generic (PLEG): container finished" podID="93da129d-b9e3-4256-aa8b-bcc6e50556f8" containerID="6023e333939437ec74f67b6fc831c2525ab44e12de524c6f76c4f5ae8630a051" exitCode=0
Oct 30 19:22:21.838862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.838813    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b" event={"ID":"93da129d-b9e3-4256-aa8b-bcc6e50556f8","Type":"ContainerDied","Data":"6023e333939437ec74f67b6fc831c2525ab44e12de524c6f76c4f5ae8630a051"}
Oct 30 19:22:21.892495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:21.892468    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd97c7a-da68-4d65-8804-a0c13ab96e37" path="/var/lib/kubelet/pods/ebd97c7a-da68-4d65-8804-a0c13ab96e37/volumes"
Oct 30 19:22:23.032599 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.032574    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:23.059079 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:23.059041    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:22:23.102894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.102818    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\") pod \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") "
Oct 30 19:22:23.103059 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.102942    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\") pod \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") "
Oct 30 19:22:23.103164 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.103066    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\" (UniqueName: \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\") pod \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\" (UID: \"93da129d-b9e3-4256-aa8b-bcc6e50556f8\") "
Oct 30 19:22:23.103737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.103690    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca" pod "93da129d-b9e3-4256-aa8b-bcc6e50556f8" (UID: "93da129d-b9e3-4256-aa8b-bcc6e50556f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:23.105596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.105569    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca" pod "93da129d-b9e3-4256-aa8b-bcc6e50556f8" (UID: "93da129d-b9e3-4256-aa8b-bcc6e50556f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:23.106040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.106015    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t" pod "93da129d-b9e3-4256-aa8b-bcc6e50556f8" (UID: "93da129d-b9e3-4256-aa8b-bcc6e50556f8"). InnerVolumeSpecName "kube-api-access-jjq2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:23.204741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.204713    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjq2t\" (UniqueName: \"kubernetes.io/projected/93da129d-b9e3-4256-aa8b-bcc6e50556f8-kube-api-access-jjq2t\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:23.204741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.204740    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:23.204894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.204750    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93da129d-b9e3-4256-aa8b-bcc6e50556f8-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:23.857613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.857583    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"
Oct 30 19:22:23.857787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.857614    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b" event={"ID":"93da129d-b9e3-4256-aa8b-bcc6e50556f8","Type":"ContainerDied","Data":"426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9"}
Oct 30 19:22:23.857787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:23.857657    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426ec211aff87b6ab829251a18d523f292d7d13dc57853221da76b6b82f232c9"
Oct 30 19:22:24.599567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.599543    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:22:24.744366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744330    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744520 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744422    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744522    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744566    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744649    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744708    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744743 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744735    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.744849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744645    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.744849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744741    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.744849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744821    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.745019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744868    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.745019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744919    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\") pod \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\" (UID: \"8e8926e2-79ba-45d0-b6d3-34e8a013b052\") "
Oct 30 19:22:24.745019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.744971    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.745593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.745508    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.745593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.745539    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.745593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.745554    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.745593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.745569    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.745593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.745589    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:24.747021 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.746994    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.747245 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.747212    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "kube-api-access-rj2s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.747365 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.747276    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.747530 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.747506    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo" pod "8e8926e2-79ba-45d0-b6d3-34e8a013b052" (UID: "8e8926e2-79ba-45d0-b6d3-34e8a013b052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:24.846538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.846465    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.846538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.846492    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.846538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.846502    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.846538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.846512    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2s5\" (UniqueName: \"kubernetes.io/projected/8e8926e2-79ba-45d0-b6d3-34e8a013b052-kube-api-access-rj2s5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.846538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.846523    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8926e2-79ba-45d0-b6d3-34e8a013b052-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.869259 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.869228    5065 generic.go:356] "Generic (PLEG): container finished" podID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerID="6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3" exitCode=0
Oct 30 19:22:24.869390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.869273    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"
Oct 30 19:22:24.869390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.869300    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" event={"ID":"8e8926e2-79ba-45d0-b6d3-34e8a013b052","Type":"ContainerDied","Data":"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"}
Oct 30 19:22:24.869390 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.869372    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf" event={"ID":"8e8926e2-79ba-45d0-b6d3-34e8a013b052","Type":"ContainerDied","Data":"e0b60c4cff84d96357b2d4ac74b4e33bcebdfb14801b5343a7773c1248836cf9"}
Oct 30 19:22:24.869528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.869394    5065 scope.go:122] "RemoveContainer" containerID="6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"
Oct 30 19:22:24.883212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.883196    5065 scope.go:122] "RemoveContainer" containerID="6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"
Oct 30 19:22:24.883541 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:24.883517    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3\": container with ID starting with 6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3 not found: ID does not exist" containerID="6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"
Oct 30 19:22:24.883628 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.883547    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3"} err="failed to get container status \"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3\": rpc error: code = NotFound desc = could not find container \"6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3\": container with ID starting with 6488a5f596898ce17beb82267a462d5320dc44bb61758e26dc1b70da3fc109f3 not found: ID does not exist"
Oct 30 19:22:24.901478 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.901457    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:22:24.906749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.906726    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-8p7lf"]
Oct 30 19:22:24.984883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:24.984851    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:25.892323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:25.892281    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" path="/var/lib/kubelet/pods/8e8926e2-79ba-45d0-b6d3-34e8a013b052/volumes"
Oct 30 19:22:28.997758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:28.997725    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:22:29.366869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:29.366820    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:22:30.113248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:30.113208    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"]
Oct 30 19:22:30.117963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:30.117928    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-84c32a-xl5pn"]
Oct 30 19:22:30.267327 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:30.265404    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:22:31.623128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:31.623071    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"]
Oct 30 19:22:31.630230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:31.630200    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d72beb-cc77b"]
Oct 30 19:22:31.889930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:31.889853    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815fe09e-d5b0-4f97-9881-9670d09eedd5" path="/var/lib/kubelet/pods/815fe09e-d5b0-4f97-9881-9670d09eedd5/volumes"
Oct 30 19:22:31.890393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:31.890377    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93da129d-b9e3-4256-aa8b-bcc6e50556f8" path="/var/lib/kubelet/pods/93da129d-b9e3-4256-aa8b-bcc6e50556f8/volumes"
Oct 30 19:22:32.854429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.854405    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 19:22:32.940472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.940440    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\") pod \"8e7a7995-a027-4057-8b41-4882ab30c912\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") "
Oct 30 19:22:32.940665 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.940476    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\" (UniqueName: \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\") pod \"8e7a7995-a027-4057-8b41-4882ab30c912\" (UID: \"8e7a7995-a027-4057-8b41-4882ab30c912\") "
Oct 30 19:22:32.940828 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.940803    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume" pod "8e7a7995-a027-4057-8b41-4882ab30c912" (UID: "8e7a7995-a027-4057-8b41-4882ab30c912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:32.941127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.941112    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7a7995-a027-4057-8b41-4882ab30c912-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:32.942844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:32.942823    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8" pod "8e7a7995-a027-4057-8b41-4882ab30c912" (UID: "8e7a7995-a027-4057-8b41-4882ab30c912"). InnerVolumeSpecName "kube-api-access-mklw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:33.022101 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.022054    5065 generic.go:356] "Generic (PLEG): container finished" podID="8e7a7995-a027-4057-8b41-4882ab30c912" containerID="d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e" exitCode=0
Oct 30 19:22:33.022216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.022118    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"
Oct 30 19:22:33.022216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.022136    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" event={"ID":"8e7a7995-a027-4057-8b41-4882ab30c912","Type":"ContainerDied","Data":"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"}
Oct 30 19:22:33.022216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.022191    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs" event={"ID":"8e7a7995-a027-4057-8b41-4882ab30c912","Type":"ContainerDied","Data":"b9cd56373b9d141b75dd4e6be1109bfc1d04a966c22c4a40ef8c6cf3909e3533"}
Oct 30 19:22:33.022216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.022212    5065 scope.go:122] "RemoveContainer" containerID="d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"
Oct 30 19:22:33.037165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.037147    5065 scope.go:122] "RemoveContainer" containerID="d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"
Oct 30 19:22:33.037447 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:33.037429    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e\": container with ID starting with d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e not found: ID does not exist" containerID="d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"
Oct 30 19:22:33.037542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.037455    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e"} err="failed to get container status \"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e\": rpc error: code = NotFound desc = could not find container \"d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e\": container with ID starting with d164f0efc6aa5bc892aea46ab81cad799fee2cb7892e562473fa1f8453d7144e not found: ID does not exist"
Oct 30 19:22:33.042741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.042716    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklw8\" (UniqueName: \"kubernetes.io/projected/8e7a7995-a027-4057-8b41-4882ab30c912-kube-api-access-mklw8\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:33.051484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.051463    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 19:22:33.059299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.059280    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-84c32a-5fc8bc46d-q5krs"]
Oct 30 19:22:33.890529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.890495    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" path="/var/lib/kubelet/pods/8e7a7995-a027-4057-8b41-4882ab30c912/volumes"
Oct 30 19:22:33.998972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:33.998934    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" podUID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" containerName="payload-processing" containerID="cri-o://e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d" gracePeriod=25
Oct 30 19:22:34.289179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.289151    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:22:34.358596 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.358569    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\") pod \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") "
Oct 30 19:22:34.358793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.358633    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\") pod \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") "
Oct 30 19:22:34.358793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.358757    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\") pod \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") "
Oct 30 19:22:34.358930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.358883    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\") pod \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\" (UID: \"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f\") "
Oct 30 19:22:34.359115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.359076    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume" pod "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" (UID: "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:34.359191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.359150    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume" pod "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" (UID: "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:34.359537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.359515    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.359635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.359544    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.361387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.361356    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt" pod "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" (UID: "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f"). InnerVolumeSpecName "kube-api-access-7xmmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:34.361487 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.361394    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials" pod "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" (UID: "e3f269a2-67c4-46ad-ab7d-9b3504d18b3f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:34.367446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.367418    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" podUID="c9f7e091-31ca-493f-b301-8b35fe6b2620" containerName="payload-processing" containerID="cri-o://716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff" gracePeriod=25
Oct 30 19:22:34.471627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.460878    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.471627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.460923    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmmt\" (UniqueName: \"kubernetes.io/projected/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f-kube-api-access-7xmmt\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.638601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.638574    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:22:34.666246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.666214    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\") pod \"c9f7e091-31ca-493f-b301-8b35fe6b2620\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") "
Oct 30 19:22:34.666383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.666359    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\") pod \"c9f7e091-31ca-493f-b301-8b35fe6b2620\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") "
Oct 30 19:22:34.666493 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.666478    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\") pod \"c9f7e091-31ca-493f-b301-8b35fe6b2620\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") "
Oct 30 19:22:34.666550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.666537    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\") pod \"c9f7e091-31ca-493f-b301-8b35fe6b2620\" (UID: \"c9f7e091-31ca-493f-b301-8b35fe6b2620\") "
Oct 30 19:22:34.667747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.667719    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume" pod "c9f7e091-31ca-493f-b301-8b35fe6b2620" (UID: "c9f7e091-31ca-493f-b301-8b35fe6b2620"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:34.667887 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.667772    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume" pod "c9f7e091-31ca-493f-b301-8b35fe6b2620" (UID: "c9f7e091-31ca-493f-b301-8b35fe6b2620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:34.669629 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.669607    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9" pod "c9f7e091-31ca-493f-b301-8b35fe6b2620" (UID: "c9f7e091-31ca-493f-b301-8b35fe6b2620"). InnerVolumeSpecName "kube-api-access-cb2z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:34.670187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.670124    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials" pod "c9f7e091-31ca-493f-b301-8b35fe6b2620" (UID: "c9f7e091-31ca-493f-b301-8b35fe6b2620"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:34.768648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.768611    5065 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-routing-overlay-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.768648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.768642    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2z9\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-kube-api-access-cb2z9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.768857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.768657    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f7e091-31ca-493f-b301-8b35fe6b2620-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:34.768857 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:34.768670    5065 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9f7e091-31ca-493f-b301-8b35fe6b2620-provider-credentials\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:35.047156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.047128    5065 generic.go:356] "Generic (PLEG): container finished" podID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" containerID="e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d" exitCode=0
Oct 30 19:22:35.047587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.047178    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"
Oct 30 19:22:35.047587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.047220    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" event={"ID":"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f","Type":"ContainerDied","Data":"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"}
Oct 30 19:22:35.047587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.047287    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4" event={"ID":"e3f269a2-67c4-46ad-ab7d-9b3504d18b3f","Type":"ContainerDied","Data":"f5797ef6bfecc33274abb0400b4c8eaf724491456b85d0294b255da4c3833ffa"}
Oct 30 19:22:35.047587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.047318    5065 scope.go:122] "RemoveContainer" containerID="e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"
Oct 30 19:22:35.049246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.049220    5065 generic.go:356] "Generic (PLEG): container finished" podID="c9f7e091-31ca-493f-b301-8b35fe6b2620" containerID="716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff" exitCode=0
Oct 30 19:22:35.049381 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.049365    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" event={"ID":"c9f7e091-31ca-493f-b301-8b35fe6b2620","Type":"ContainerDied","Data":"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"}
Oct 30 19:22:35.049430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.049411    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn" event={"ID":"c9f7e091-31ca-493f-b301-8b35fe6b2620","Type":"ContainerDied","Data":"6871e51754f7cb0da4457a3dc3ca2b6b1442b55885a43d5fe9f90cb681a929b5"}
Oct 30 19:22:35.049491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.049480    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"
Oct 30 19:22:35.059303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.059285    5065 scope.go:122] "RemoveContainer" containerID="e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"
Oct 30 19:22:35.059577 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:35.059558    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d\": container with ID starting with e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d not found: ID does not exist" containerID="e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"
Oct 30 19:22:35.059654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.059583    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d"} err="failed to get container status \"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d\": rpc error: code = NotFound desc = could not find container \"e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d\": container with ID starting with e3b0bcc1b0d32e966970288d9238ecd438340f8ba762c41e17e25f63ca6a035d not found: ID does not exist"
Oct 30 19:22:35.059654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.059600    5065 scope.go:122] "RemoveContainer" containerID="716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"
Oct 30 19:22:35.069227 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.069210    5065 scope.go:122] "RemoveContainer" containerID="716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"
Oct 30 19:22:35.069497 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:35.069476    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff\": container with ID starting with 716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff not found: ID does not exist" containerID="716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"
Oct 30 19:22:35.069547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.069505    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff"} err="failed to get container status \"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff\": rpc error: code = NotFound desc = could not find container \"716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff\": container with ID starting with 716b5c55dd10e7d9a5c4278b0c1f8d9cb9133976d77565eb1624fbac9f98e5ff not found: ID does not exist"
Oct 30 19:22:35.077465 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.077443    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:22:35.087895 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.087870    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-76c59jsm4"]
Oct 30 19:22:35.097922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.097900    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:22:35.102333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.102308    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-79b7tvwqn"]
Oct 30 19:22:35.268413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.268352    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing" containerID="cri-o://8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52" gracePeriod=25
Oct 30 19:22:35.891922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.891888    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f7e091-31ca-493f-b301-8b35fe6b2620" path="/var/lib/kubelet/pods/c9f7e091-31ca-493f-b301-8b35fe6b2620/volumes"
Oct 30 19:22:35.892441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:35.892423    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" path="/var/lib/kubelet/pods/e3f269a2-67c4-46ad-ab7d-9b3504d18b3f/volumes"
Oct 30 19:22:40.275243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:40.275210    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 19:22:40.275796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:40.275423    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy" containerID="cri-o://6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3" gracePeriod=30
Oct 30 19:22:41.715148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:41.715112    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:22:41.715731 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:41.715453    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" containerID="cri-o://26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82" gracePeriod=30
Oct 30 19:22:41.725504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:41.725479    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:22:41.725771 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:41.725697    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy" containerID="cri-o://701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562" gracePeriod=30
Oct 30 19:22:42.298538 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.298499    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"]
Oct 30 19:22:42.309863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.309834    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" containerName="payload-processing"
Oct 30 19:22:42.310114 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.310073    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" containerName="payload-processing"
Oct 30 19:22:42.310299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.310285    5065 cpu_manager.go:414] "containerMap: removing container" podUID="93da129d-b9e3-4256-aa8b-bcc6e50556f8" containerName="revoke-keys"
Oct 30 19:22:42.314279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314255    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93da129d-b9e3-4256-aa8b-bcc6e50556f8" containerName="revoke-keys"
Oct 30 19:22:42.314449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314435    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing"
Oct 30 19:22:42.314565 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314551    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing"
Oct 30 19:22:42.314662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314652    5065 cpu_manager.go:414] "containerMap: removing container" podUID="27d4e518-56e0-4938-bc8e-63a380297efa" containerName="revoke-keys"
Oct 30 19:22:42.314759 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314748    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d4e518-56e0-4938-bc8e-63a380297efa" containerName="revoke-keys"
Oct 30 19:22:42.314865 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314856    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy"
Oct 30 19:22:42.314939 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.314929    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy"
Oct 30 19:22:42.315028 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.315019    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c9f7e091-31ca-493f-b301-8b35fe6b2620" containerName="payload-processing"
Oct 30 19:22:42.315165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.315153    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f7e091-31ca-493f-b301-8b35fe6b2620" containerName="payload-processing"
Oct 30 19:22:42.315469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.315450    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f269a2-67c4-46ad-ab7d-9b3504d18b3f" containerName="payload-processing"
Oct 30 19:22:42.315611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.315599    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="93da129d-b9e3-4256-aa8b-bcc6e50556f8" containerName="revoke-keys"
Oct 30 19:22:42.315732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.315721    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7a7995-a027-4057-8b41-4882ab30c912" containerName="payload-processing"
Oct 30 19:22:42.332269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.332235    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d4e518-56e0-4938-bc8e-63a380297efa" containerName="revoke-keys"
Oct 30 19:22:42.332597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.332568    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8926e2-79ba-45d0-b6d3-34e8a013b052" containerName="istio-proxy"
Oct 30 19:22:42.332716 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.332705    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f7e091-31ca-493f-b301-8b35fe6b2620" containerName="payload-processing"
Oct 30 19:22:42.385146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.383669    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.385146 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.383565    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"]
Oct 30 19:22:42.390807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.390103    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:22:42.390807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.390170    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:42.462850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.462747    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdjx\" (UniqueName: \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.462850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.462857    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.463552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.463127    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.569166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.568182    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.569166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.568525    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.569166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.568590    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdjx\" (UniqueName: \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.570223 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.570193    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.587737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.587692    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.604050 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.604018    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdjx\" (UniqueName: \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:42.722205 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:42.722170    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:43.074243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:43.074214    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"]
Oct 30 19:22:43.074864 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:22:43.074839    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1439533_e4e4_4381_b09e_8d0fab1a55b3.slice/crio-c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061 WatchSource:0}: Error finding container c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061: Status 404 returned error can't find the container with id c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061
Oct 30 19:22:43.159351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:43.159322    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk" event={"ID":"e1439533-e4e4-4381-b09e-8d0fab1a55b3","Type":"ContainerStarted","Data":"c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061"}
Oct 30 19:22:44.169400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:44.169369    5065 generic.go:356] "Generic (PLEG): container finished" podID="e1439533-e4e4-4381-b09e-8d0fab1a55b3" containerID="e8bfb8b17b3716f2a3e098c15c01ed1ca40a2d0c5c06340b0f08b1e678d3793d" exitCode=0
Oct 30 19:22:44.169778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:44.169463    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk" event={"ID":"e1439533-e4e4-4381-b09e-8d0fab1a55b3","Type":"ContainerDied","Data":"e8bfb8b17b3716f2a3e098c15c01ed1ca40a2d0c5c06340b0f08b1e678d3793d"}
Oct 30 19:22:44.778592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:44.778549    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:45.363954 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.363920    5065 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:45.364316 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.363982    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:45.371656 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.371637    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:45.509545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.509521    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\") pod \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") "
Oct 30 19:22:45.509722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.509581    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\") pod \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") "
Oct 30 19:22:45.509722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.509606    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\" (UniqueName: \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\") pod \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\" (UID: \"e1439533-e4e4-4381-b09e-8d0fab1a55b3\") "
Oct 30 19:22:45.509910 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.509884    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca" pod "e1439533-e4e4-4381-b09e-8d0fab1a55b3" (UID: "e1439533-e4e4-4381-b09e-8d0fab1a55b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:45.510173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.510153    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.512135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.512107    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx" pod "e1439533-e4e4-4381-b09e-8d0fab1a55b3" (UID: "e1439533-e4e4-4381-b09e-8d0fab1a55b3"). InnerVolumeSpecName "kube-api-access-kpdjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:45.512240 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.512168    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca" pod "e1439533-e4e4-4381-b09e-8d0fab1a55b3" (UID: "e1439533-e4e4-4381-b09e-8d0fab1a55b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:45.611475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.611430    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1439533-e4e4-4381-b09e-8d0fab1a55b3-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.611475 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.611459    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdjx\" (UniqueName: \"kubernetes.io/projected/e1439533-e4e4-4381-b09e-8d0fab1a55b3-kube-api-access-kpdjx\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.970022 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:45.969989    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 19:22:46.119544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119461    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.119544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119537    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.119777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119610    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.119777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119633    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.119777 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119664    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119742    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119852    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119892    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.119941    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.120154    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.120230    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:46.120366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.120262    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\") pod \"c4a57750-7509-4b7b-b609-0d161d72f037\" (UID: \"c4a57750-7509-4b7b-b609-0d161d72f037\") "
Oct 30 19:22:46.121056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.120482    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:46.121056 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.120555    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:46.121414 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.121377    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.121414 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.121404    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a57750-7509-4b7b-b609-0d161d72f037-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.121414 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.121420    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.122341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.121437    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.122341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.121451    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.122592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.122569    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "kube-api-access-5cptq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:46.122677 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.122608    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:46.123073 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.123047    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:46.123476 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.123443    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy" pod "c4a57750-7509-4b7b-b609-0d161d72f037" (UID: "c4a57750-7509-4b7b-b609-0d161d72f037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:46.187575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.187542    5065 generic.go:356] "Generic (PLEG): container finished" podID="c4a57750-7509-4b7b-b609-0d161d72f037" containerID="6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3" exitCode=0
Oct 30 19:22:46.187724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.187603    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"
Oct 30 19:22:46.187724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.187631    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" event={"ID":"c4a57750-7509-4b7b-b609-0d161d72f037","Type":"ContainerDied","Data":"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"}
Oct 30 19:22:46.187844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.187727    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr" event={"ID":"c4a57750-7509-4b7b-b609-0d161d72f037","Type":"ContainerDied","Data":"05e7f2d8ac2aaae000f55bcab6d4f37eef1ab577f397d78d7065f65e1c6d8cad"}
Oct 30 19:22:46.187844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.187762    5065 scope.go:122] "RemoveContainer" containerID="6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"
Oct 30 19:22:46.189408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.189383    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"
Oct 30 19:22:46.189536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.189387    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk" event={"ID":"e1439533-e4e4-4381-b09e-8d0fab1a55b3","Type":"ContainerDied","Data":"c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061"}
Oct 30 19:22:46.189536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.189490    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4417050093b9fc2294694c1db54d148fcb1974f373d65ff7b94213e35a4a061"
Oct 30 19:22:46.200561 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.200543    5065 scope.go:122] "RemoveContainer" containerID="6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"
Oct 30 19:22:46.200821 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:46.200802    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3\": container with ID starting with 6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3 not found: ID does not exist" containerID="6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"
Oct 30 19:22:46.200878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.200838    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3"} err="failed to get container status \"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3\": rpc error: code = NotFound desc = could not find container \"6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3\": container with ID starting with 6fef1049e922e939d320ecd44a9880ac53014718a966414c5cddc46d56616df3 not found: ID does not exist"
Oct 30 19:22:46.222837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.222813    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a57750-7509-4b7b-b609-0d161d72f037-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.222837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.222835    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cptq\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-kube-api-access-5cptq\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.222974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.222847    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a57750-7509-4b7b-b609-0d161d72f037-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.222974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.222856    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a57750-7509-4b7b-b609-0d161d72f037-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.225875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.225853    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 19:22:46.232581 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:46.232559    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-8skzr"]
Oct 30 19:22:47.042883 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.042857    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:22:47.063031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.063005    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:22:47.136652 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136615    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.136834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136684    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.136834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136746    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.136834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136779    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.136834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136824    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.137055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136898    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.137055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.136918    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.137055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137037    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.137278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137103    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.137278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137234    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\") pod \"5dbafd20-cb17-4156-ac4f-360d7649461f\" (UID: \"5dbafd20-cb17-4156-ac4f-360d7649461f\") "
Oct 30 19:22:47.137385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137316    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.137544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137519    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:47.137663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137642    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.137817 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137791    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.137878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137825    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.137878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137835    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.137878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137842    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbafd20-cb17-4156-ac4f-360d7649461f-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.138000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.137878    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.139831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.139800    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.140284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.140264    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:47.140514 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.140492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.140831 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.140811    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6" pod "5dbafd20-cb17-4156-ac4f-360d7649461f" (UID: "5dbafd20-cb17-4156-ac4f-360d7649461f"). InnerVolumeSpecName "kube-api-access-5cmw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.200689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.200662    5065 generic.go:356] "Generic (PLEG): container finished" podID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerID="701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562" exitCode=0
Oct 30 19:22:47.200800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.200711    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"
Oct 30 19:22:47.200800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.200752    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" event={"ID":"ba32d319-54ac-4975-bf83-35d51f752fd1","Type":"ContainerDied","Data":"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"}
Oct 30 19:22:47.200914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.200809    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr" event={"ID":"ba32d319-54ac-4975-bf83-35d51f752fd1","Type":"ContainerDied","Data":"650a079532573dde66c735af7fe7c4413d686f1964f3397f4a03907894bf6fa4"}
Oct 30 19:22:47.200914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.200842    5065 scope.go:122] "RemoveContainer" containerID="701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"
Oct 30 19:22:47.202537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.202514    5065 generic.go:356] "Generic (PLEG): container finished" podID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerID="26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82" exitCode=0
Oct 30 19:22:47.202695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.202544    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" event={"ID":"5dbafd20-cb17-4156-ac4f-360d7649461f","Type":"ContainerDied","Data":"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"}
Oct 30 19:22:47.202695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.202578    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"
Oct 30 19:22:47.202695 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.202598    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2" event={"ID":"5dbafd20-cb17-4156-ac4f-360d7649461f","Type":"ContainerDied","Data":"512be5fcb5d17e96e25f55100531679d1197ab9811ccc574fbd52924e38227df"}
Oct 30 19:22:47.213014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.212996    5065 scope.go:122] "RemoveContainer" containerID="701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"
Oct 30 19:22:47.213301 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:47.213282    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562\": container with ID starting with 701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562 not found: ID does not exist" containerID="701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"
Oct 30 19:22:47.213371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.213308    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562"} err="failed to get container status \"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562\": rpc error: code = NotFound desc = could not find container \"701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562\": container with ID starting with 701fca8f3dab6e0ac4fc66845db1ec02d6358800dab8374d25bf8f915aea5562 not found: ID does not exist"
Oct 30 19:22:47.213371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.213325    5065 scope.go:122] "RemoveContainer" containerID="26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"
Oct 30 19:22:47.224186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.224169    5065 scope.go:122] "RemoveContainer" containerID="26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"
Oct 30 19:22:47.224489 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:47.224472    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82\": container with ID starting with 26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82 not found: ID does not exist" containerID="26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"
Oct 30 19:22:47.224541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.224495    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82"} err="failed to get container status \"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82\": rpc error: code = NotFound desc = could not find container \"26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82\": container with ID starting with 26c68a2631fe27fbf6efa3aae6a8d7738006333fa5b1fd4e95aaf7d7a35a3c82 not found: ID does not exist"
Oct 30 19:22:47.238822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.238800    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.238930 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.238871    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.238992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.238926    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.238992 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.238962    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239014    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239158    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.239196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239171    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239202    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239304    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.239361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239320    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239331    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:47.239526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239376    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\") pod \"ba32d319-54ac-4975-bf83-35d51f752fd1\" (UID: \"ba32d319-54ac-4975-bf83-35d51f752fd1\") "
Oct 30 19:22:47.239526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239399    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.239667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239648    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.239835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239816    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239841    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239856    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32d319-54ac-4975-bf83-35d51f752fd1-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239869    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239877    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239886    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239901    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.239914 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239912    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.240540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239926    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.240540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.239941    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cmw6\" (UniqueName: \"kubernetes.io/projected/5dbafd20-cb17-4156-ac4f-360d7649461f-kube-api-access-5cmw6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.241382 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.241362    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "kube-api-access-xmfsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.241654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.241640    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.241714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.241691    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:47.241753 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.241708    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token" pod "ba32d319-54ac-4975-bf83-35d51f752fd1" (UID: "ba32d319-54ac-4975-bf83-35d51f752fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.263881 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.263832    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:22:47.269861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.269842    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-nk2j2"]
Oct 30 19:22:47.341459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.341434    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfsk\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-kube-api-access-xmfsk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.341459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.341457    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.341589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.341467    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.341589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.341476    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32d319-54ac-4975-bf83-35d51f752fd1-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.567168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.564565    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:22:47.571160 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.571080    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-pk5nr"]
Oct 30 19:22:47.893176 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.893073    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" path="/var/lib/kubelet/pods/5dbafd20-cb17-4156-ac4f-360d7649461f/volumes"
Oct 30 19:22:47.893652 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.893629    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" path="/var/lib/kubelet/pods/ba32d319-54ac-4975-bf83-35d51f752fd1/volumes"
Oct 30 19:22:47.894137 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:47.894118    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" path="/var/lib/kubelet/pods/c4a57750-7509-4b7b-b609-0d161d72f037/volumes"
Oct 30 19:22:53.059040 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:53.059001    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:22:54.510580 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:54.510541    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"]
Oct 30 19:22:54.517543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:54.517508    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a962a-2m7zk"]
Oct 30 19:22:54.778579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:54.778491    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:55.604540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.604510    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:22:55.608613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.608576    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\") pod \"8521b31a-c79c-45aa-84bf-47a618e04f3d\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") "
Oct 30 19:22:55.608613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.608614    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\" (UniqueName: \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\") pod \"8521b31a-c79c-45aa-84bf-47a618e04f3d\" (UID: \"8521b31a-c79c-45aa-84bf-47a618e04f3d\") "
Oct 30 19:22:55.609012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.608985    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume" pod "8521b31a-c79c-45aa-84bf-47a618e04f3d" (UID: "8521b31a-c79c-45aa-84bf-47a618e04f3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:55.610864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.610841    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph" pod "8521b31a-c79c-45aa-84bf-47a618e04f3d" (UID: "8521b31a-c79c-45aa-84bf-47a618e04f3d"). InnerVolumeSpecName "kube-api-access-wkjph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:55.709860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.709826    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8521b31a-c79c-45aa-84bf-47a618e04f3d-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.709860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.709854    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkjph\" (UniqueName: \"kubernetes.io/projected/8521b31a-c79c-45aa-84bf-47a618e04f3d-kube-api-access-wkjph\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.890747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:55.890659    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1439533-e4e4-4381-b09e-8d0fab1a55b3" path="/var/lib/kubelet/pods/e1439533-e4e4-4381-b09e-8d0fab1a55b3/volumes"
Oct 30 19:22:56.288439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.288403    5065 generic.go:356] "Generic (PLEG): container finished" podID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerID="8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52" exitCode=0
Oct 30 19:22:56.288680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.288450    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"
Oct 30 19:22:56.288680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.288486    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" event={"ID":"8521b31a-c79c-45aa-84bf-47a618e04f3d","Type":"ContainerDied","Data":"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"}
Oct 30 19:22:56.288680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.288541    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s" event={"ID":"8521b31a-c79c-45aa-84bf-47a618e04f3d","Type":"ContainerDied","Data":"17423a66c3f4425b50010f4a40eb9042b4ba8d0de3ea600fe6707bc5990cf346"}
Oct 30 19:22:56.288680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.288563    5065 scope.go:122] "RemoveContainer" containerID="8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"
Oct 30 19:22:56.300718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.300702    5065 scope.go:122] "RemoveContainer" containerID="8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"
Oct 30 19:22:56.301045 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:22:56.301026    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52\": container with ID starting with 8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52 not found: ID does not exist" containerID="8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"
Oct 30 19:22:56.301147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.301053    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52"} err="failed to get container status \"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52\": rpc error: code = NotFound desc = could not find container \"8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52\": container with ID starting with 8b5998731c5c0511f5109c2726515fe592cd47d2832adf47e8c25a8f07a9ac52 not found: ID does not exist"
Oct 30 19:22:56.314115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.314063    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:22:56.318651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:56.318631    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d72beb-5f695cc65d-2l84s"]
Oct 30 19:22:57.890805 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:22:57.890772    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" path="/var/lib/kubelet/pods/8521b31a-c79c-45aa-84bf-47a618e04f3d/volumes"
Oct 30 19:23:02.564890 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:02.564647    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:23:02.565319 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:02.564900    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy" containerID="cri-o://7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2" gracePeriod=30
Oct 30 19:23:03.102071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102031    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"]
Oct 30 19:23:03.102692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102675    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing"
Oct 30 19:23:03.102762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102697    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing"
Oct 30 19:23:03.102762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102721    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy"
Oct 30 19:23:03.102762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102726    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy"
Oct 30 19:23:03.102762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102756    5065 cpu_manager.go:414] "containerMap: removing container" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy"
Oct 30 19:23:03.102762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102761    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102777    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e1439533-e4e4-4381-b09e-8d0fab1a55b3" containerName="revoke-keys"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102785    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1439533-e4e4-4381-b09e-8d0fab1a55b3" containerName="revoke-keys"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102810    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102814    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102901    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dbafd20-cb17-4156-ac4f-360d7649461f" containerName="istio-proxy"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102910    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1439533-e4e4-4381-b09e-8d0fab1a55b3" containerName="revoke-keys"
Oct 30 19:23:03.102922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102921    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a57750-7509-4b7b-b609-0d161d72f037" containerName="istio-proxy"
Oct 30 19:23:03.103167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102933    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8521b31a-c79c-45aa-84bf-47a618e04f3d" containerName="payload-processing"
Oct 30 19:23:03.103167 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.102944    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba32d319-54ac-4975-bf83-35d51f752fd1" containerName="istio-proxy"
Oct 30 19:23:03.106749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.106733    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.116462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.116436    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:03.119885 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.119861    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.120027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.119954    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5h69\" (UniqueName: \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.120027 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.119983    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.120341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.120319    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"]
Oct 30 19:23:03.214801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.214773    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:23:03.221489 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.221459    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.221635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.221581    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5h69\" (UniqueName: \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.221635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.221621    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.222344 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.222325    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.223988 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.223971    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.236613 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.236585    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5h69\" (UniqueName: \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.424024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.423943    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:03.730730 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:23:03.730689    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee17a9e_c677_4c77_a64e_110c382889b8.slice/crio-307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9 WatchSource:0}: Error finding container 307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9: Status 404 returned error can't find the container with id 307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9
Oct 30 19:23:03.731219 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:03.730858    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"]
Oct 30 19:23:04.004755 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.004670    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:23:04.012715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.012688    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:23:04.052144 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.052124    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:23:04.059926 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.059907    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:23:04.371145 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.371104    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh" event={"ID":"8ee17a9e-c677-4c77-a64e-110c382889b8","Type":"ContainerStarted","Data":"d82aa4d525a626d7a8865503c81c2b94382b0801418a6ee10abaeda8306d2808"}
Oct 30 19:23:04.371329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.371179    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh" event={"ID":"8ee17a9e-c677-4c77-a64e-110c382889b8","Type":"ContainerStarted","Data":"307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9"}
Oct 30 19:23:04.417643 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:04.417581    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh" podStartSLOduration=1.417565004 podStartE2EDuration="1.417565004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:04.416869992 +0000 UTC m=+3004.680334956" watchObservedRunningTime="2026-10-30 19:23:04.417565004 +0000 UTC m=+3004.681029951"
Oct 30 19:23:05.381289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:05.381249    5065 generic.go:356] "Generic (PLEG): container finished" podID="8ee17a9e-c677-4c77-a64e-110c382889b8" containerID="d82aa4d525a626d7a8865503c81c2b94382b0801418a6ee10abaeda8306d2808" exitCode=0
Oct 30 19:23:05.381789 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:05.381335    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh" event={"ID":"8ee17a9e-c677-4c77-a64e-110c382889b8","Type":"ContainerDied","Data":"d82aa4d525a626d7a8865503c81c2b94382b0801418a6ee10abaeda8306d2808"}
Oct 30 19:23:06.552736 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.552713    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:06.590546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.590518    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\") pod \"8ee17a9e-c677-4c77-a64e-110c382889b8\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") "
Oct 30 19:23:06.590715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.590691    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\" (UniqueName: \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\") pod \"8ee17a9e-c677-4c77-a64e-110c382889b8\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") "
Oct 30 19:23:06.590715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.590744    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\") pod \"8ee17a9e-c677-4c77-a64e-110c382889b8\" (UID: \"8ee17a9e-c677-4c77-a64e-110c382889b8\") "
Oct 30 19:23:06.591858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.591832    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca" pod "8ee17a9e-c677-4c77-a64e-110c382889b8" (UID: "8ee17a9e-c677-4c77-a64e-110c382889b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:06.593024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.592998    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca" pod "8ee17a9e-c677-4c77-a64e-110c382889b8" (UID: "8ee17a9e-c677-4c77-a64e-110c382889b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:06.594041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.594016    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69" pod "8ee17a9e-c677-4c77-a64e-110c382889b8" (UID: "8ee17a9e-c677-4c77-a64e-110c382889b8"). InnerVolumeSpecName "kube-api-access-q5h69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:06.692668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.692585    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.692668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.692617    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5h69\" (UniqueName: \"kubernetes.io/projected/8ee17a9e-c677-4c77-a64e-110c382889b8-kube-api-access-q5h69\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:06.692668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:06.692631    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee17a9e-c677-4c77-a64e-110c382889b8-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.399816 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.399787    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"
Oct 30 19:23:07.399990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.399814    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh" event={"ID":"8ee17a9e-c677-4c77-a64e-110c382889b8","Type":"ContainerDied","Data":"307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9"}
Oct 30 19:23:07.399990 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.399859    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307caa7b2ec74981cf406c268776d4686b7549142bcbb00b77296f258a3ebdc9"
Oct 30 19:23:07.854023 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.854003    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:23:07.911770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.911738    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.911951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.911834    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.911951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.911892    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.911960    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912007    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.912115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912074    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912139    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912150    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.912296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912257    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912295    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912469 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912411    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\") pod \"ac748607-954a-4ae1-9803-467c3458b486\" (UID: \"ac748607-954a-4ae1-9803-467c3458b486\") "
Oct 30 19:23:07.912696 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912673    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.912873 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912828    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.912967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912940    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.913014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912975    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.913014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.912991    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.913014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.913006    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.913373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.913333    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:07.916329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.916249    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:23:07.916329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.916294    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:07.916526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.916506    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.917409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:07.917388    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc" pod "ac748607-954a-4ae1-9803-467c3458b486" (UID: "ac748607-954a-4ae1-9803-467c3458b486"). InnerVolumeSpecName "kube-api-access-s84sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:08.013878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.013837    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac748607-954a-4ae1-9803-467c3458b486-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.013878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.013871    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84sc\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-kube-api-access-s84sc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.014107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.013887    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac748607-954a-4ae1-9803-467c3458b486-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.014107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.013901    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac748607-954a-4ae1-9803-467c3458b486-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.014107 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.013916    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac748607-954a-4ae1-9803-467c3458b486-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.425779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.425685    5065 generic.go:356] "Generic (PLEG): container finished" podID="ac748607-954a-4ae1-9803-467c3458b486" containerID="7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2" exitCode=0
Oct 30 19:23:08.425967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.425913    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" event={"ID":"ac748607-954a-4ae1-9803-467c3458b486","Type":"ContainerDied","Data":"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"}
Oct 30 19:23:08.426047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.426001    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk" event={"ID":"ac748607-954a-4ae1-9803-467c3458b486","Type":"ContainerDied","Data":"4cff79c2f6dfbc1cb7ec154c1cc11053e9a76496a16e2a0df77ad81b7359aea7"}
Oct 30 19:23:08.426047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.426036    5065 scope.go:122] "RemoveContainer" containerID="7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"
Oct 30 19:23:08.426265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.426245    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"
Oct 30 19:23:08.446804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.446770    5065 scope.go:122] "RemoveContainer" containerID="7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"
Oct 30 19:23:08.447139 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:08.447113    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2\": container with ID starting with 7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2 not found: ID does not exist" containerID="7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"
Oct 30 19:23:08.447260 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.447142    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2"} err="failed to get container status \"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2\": rpc error: code = NotFound desc = could not find container \"7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2\": container with ID starting with 7ae517b6ba460eba59dedabc90d05accca8cd7c0e873c483acefec63a8faccd2 not found: ID does not exist"
Oct 30 19:23:08.489526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.489497    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:23:08.493246 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:08.493227    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kl8xk"]
Oct 30 19:23:09.892285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:09.892252    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac748607-954a-4ae1-9803-467c3458b486" path="/var/lib/kubelet/pods/ac748607-954a-4ae1-9803-467c3458b486/volumes"
Oct 30 19:23:13.206243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:13.206153    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 19:23:13.216102 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:13.216022    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 19:23:16.255324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:16.255289    5065 scope.go:122] "RemoveContainer" containerID="61858fa3e539a964de4c763a9d70f37432f22518554735a672be355dac48154b"
Oct 30 19:23:16.914752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:16.914722    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"]
Oct 30 19:23:16.918401 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:16.918374    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ab7df0-9w9bh"]
Oct 30 19:23:17.892489 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:17.892459    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee17a9e-c677-4c77-a64e-110c382889b8" path="/var/lib/kubelet/pods/8ee17a9e-c677-4c77-a64e-110c382889b8/volumes"
Oct 30 19:23:18.210048 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:18.207526    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing" containerID="cri-o://9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d" gracePeriod=25
Oct 30 19:23:18.218218 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:18.218149    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing" containerID="cri-o://699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844" gracePeriod=25
Oct 30 19:23:23.058739 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:23.058704    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:23:24.330129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:24.327794    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:24.340181 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:24.336467    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:26.146108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.146062    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 19:23:26.146833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.146801    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy" containerID="cri-o://ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a" gracePeriod=30
Oct 30 19:23:26.855104 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855053    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"]
Oct 30 19:23:26.855780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855754    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8ee17a9e-c677-4c77-a64e-110c382889b8" containerName="revoke-keys"
Oct 30 19:23:26.855780 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855775    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee17a9e-c677-4c77-a64e-110c382889b8" containerName="revoke-keys"
Oct 30 19:23:26.855993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855803    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy"
Oct 30 19:23:26.855993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855810    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy"
Oct 30 19:23:26.855993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855925    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee17a9e-c677-4c77-a64e-110c382889b8" containerName="revoke-keys"
Oct 30 19:23:26.855993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.855939    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac748607-954a-4ae1-9803-467c3458b486" containerName="istio-proxy"
Oct 30 19:23:26.861872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.861853    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:26.868845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.868816    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:23:26.869018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.868831    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:26.875128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.871141    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"]
Oct 30 19:23:26.999011 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.998973    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:26.999220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.999166    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:26.999300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:26.999241    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lct7q\" (UniqueName: \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.100351 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.100318    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.100558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.100456    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.100558 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.100543    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lct7q\" (UniqueName: \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.102033 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.101999    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.105553 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.105485    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.115363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.115340    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lct7q\" (UniqueName: \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\") pod \"maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.190793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.190775    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:27.479356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.479318    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"]
Oct 30 19:23:27.479688 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:23:27.479667    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffffacd1_2acc_4718_b3a0_79e09638f689.slice/crio-26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed WatchSource:0}: Error finding container 26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed: Status 404 returned error can't find the container with id 26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed
Oct 30 19:23:27.598173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:27.598138    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j" event={"ID":"ffffacd1-2acc-4718-b3a0-79e09638f689","Type":"ContainerStarted","Data":"26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed"}
Oct 30 19:23:28.608611 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:28.608577    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j" event={"ID":"ffffacd1-2acc-4718-b3a0-79e09638f689","Type":"ContainerStarted","Data":"f9a87c56722be9f8fcdbd4ce82687873f8e43977ad52babb098a1126280c5f40"}
Oct 30 19:23:28.657779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:28.657730    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j" podStartSLOduration=2.657715958 podStartE2EDuration="2.657715958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:28.653768479 +0000 UTC m=+3028.917233472" watchObservedRunningTime="2026-10-30 19:23:28.657715958 +0000 UTC m=+3028.921180906"
Oct 30 19:23:28.765383 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:28.765347    5065 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffffacd1_2acc_4718_b3a0_79e09638f689.slice/crio-conmon-f9a87c56722be9f8fcdbd4ce82687873f8e43977ad52babb098a1126280c5f40.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:23:29.618980 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:29.618944    5065 generic.go:356] "Generic (PLEG): container finished" podID="ffffacd1-2acc-4718-b3a0-79e09638f689" containerID="f9a87c56722be9f8fcdbd4ce82687873f8e43977ad52babb098a1126280c5f40" exitCode=0
Oct 30 19:23:29.619450 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:29.619054    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j" event={"ID":"ffffacd1-2acc-4718-b3a0-79e09638f689","Type":"ContainerDied","Data":"f9a87c56722be9f8fcdbd4ce82687873f8e43977ad52babb098a1126280c5f40"}
Oct 30 19:23:30.784582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.784555    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:30.842983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.842953    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\" (UniqueName: \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\") pod \"ffffacd1-2acc-4718-b3a0-79e09638f689\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") "
Oct 30 19:23:30.843195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.843105    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\") pod \"ffffacd1-2acc-4718-b3a0-79e09638f689\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") "
Oct 30 19:23:30.843271 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.843201    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\") pod \"ffffacd1-2acc-4718-b3a0-79e09638f689\" (UID: \"ffffacd1-2acc-4718-b3a0-79e09638f689\") "
Oct 30 19:23:30.843507 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.843481    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca" pod "ffffacd1-2acc-4718-b3a0-79e09638f689" (UID: "ffffacd1-2acc-4718-b3a0-79e09638f689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:30.843669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.843648    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.845346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.845318    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q" pod "ffffacd1-2acc-4718-b3a0-79e09638f689" (UID: "ffffacd1-2acc-4718-b3a0-79e09638f689"). InnerVolumeSpecName "kube-api-access-lct7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:30.845446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.845344    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca" pod "ffffacd1-2acc-4718-b3a0-79e09638f689" (UID: "ffffacd1-2acc-4718-b3a0-79e09638f689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:30.945243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.945179    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffffacd1-2acc-4718-b3a0-79e09638f689-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.945243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:30.945207    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lct7q\" (UniqueName: \"kubernetes.io/projected/ffffacd1-2acc-4718-b3a0-79e09638f689-kube-api-access-lct7q\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.435186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.435159    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 19:23:31.453563 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453539    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453590    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453620    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453695    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453736    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453790    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:31.453909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453810    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.453909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453887    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.454155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453925    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.454155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.453974    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\") pod \"b80abb5f-43ce-4af7-956a-54165637a734\" (UID: \"b80abb5f-43ce-4af7-956a-54165637a734\") "
Oct 30 19:23:31.454155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454016    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:31.454314 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454268    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:31.454385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454359    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:31.454385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454362    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:31.454701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454680    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.454766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454708    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.454766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454725    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.454766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454739    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.454766 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.454753    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80abb5f-43ce-4af7-956a-54165637a734-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.456026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.456000    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "kube-api-access-ltsq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:31.456186 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.456060    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:23:31.456537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.456513    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:31.456621 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.456583    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token" pod "b80abb5f-43ce-4af7-956a-54165637a734" (UID: "b80abb5f-43ce-4af7-956a-54165637a734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:31.555505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.555399    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80abb5f-43ce-4af7-956a-54165637a734-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.555505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.555427    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsq2\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-kube-api-access-ltsq2\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.555505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.555438    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80abb5f-43ce-4af7-956a-54165637a734-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.555505 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.555448    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80abb5f-43ce-4af7-956a-54165637a734-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:31.636371 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.636337    5065 generic.go:356] "Generic (PLEG): container finished" podID="b80abb5f-43ce-4af7-956a-54165637a734" containerID="ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a" exitCode=0
Oct 30 19:23:31.636543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.636442    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"
Oct 30 19:23:31.636543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.636476    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerDied","Data":"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"}
Oct 30 19:23:31.636543 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.636533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7" event={"ID":"b80abb5f-43ce-4af7-956a-54165637a734","Type":"ContainerDied","Data":"0c343063c04b5a22174be9df45336c52c65f961fd4383bbce1924e3bd63fe0e7"}
Oct 30 19:23:31.636688 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.636562    5065 scope.go:122] "RemoveContainer" containerID="ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"
Oct 30 19:23:31.638119 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.638078    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"
Oct 30 19:23:31.638252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.638133    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j" event={"ID":"ffffacd1-2acc-4718-b3a0-79e09638f689","Type":"ContainerDied","Data":"26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed"}
Oct 30 19:23:31.638252 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.638178    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26274cc4b73d0ae58170a21f4322d3089b37a2ef2600c9e259bc652adb0c52ed"
Oct 30 19:23:31.649966 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.649947    5065 scope.go:122] "RemoveContainer" containerID="06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.662016    5065 scope.go:122] "RemoveContainer" containerID="ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:31.662507    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a\": container with ID starting with ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a not found: ID does not exist" containerID="ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.662540    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a"} err="failed to get container status \"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a\": rpc error: code = NotFound desc = could not find container \"ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a\": container with ID starting with ed7d1fe90bb59a1bae1d46027fa48336d92269eeee91d3f349ac63bc7281797a not found: ID does not exist"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.662563    5065 scope.go:122] "RemoveContainer" containerID="06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:31.663039    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939\": container with ID starting with 06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939 not found: ID does not exist" containerID="06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"
Oct 30 19:23:31.666210 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.663141    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939"} err="failed to get container status \"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939\": rpc error: code = NotFound desc = could not find container \"06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939\": container with ID starting with 06a33377d96b826c51f9a2b7d46b167c3ca19a51f45214354fc5243097fc2939 not found: ID does not exist"
Oct 30 19:23:31.680859 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.680839    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 19:23:31.694633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.694338    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ab7df0-openshift-default-56574f6b4c-s7tc7"]
Oct 30 19:23:31.891049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:31.890950    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80abb5f-43ce-4af7-956a-54165637a734" path="/var/lib/kubelet/pods/b80abb5f-43ce-4af7-956a-54165637a734/volumes"
Oct 30 19:23:34.319572 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:34.319530    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:34.320024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:34.319543    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:38.574648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.574613    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 19:23:38.606820 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.606787    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 19:23:38.654109 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654060    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\") pod \"bc364195-6c9a-4902-89a5-b2150d1f68b7\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") "
Oct 30 19:23:38.654310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654151    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\") pod \"893e6079-0241-4e01-a668-47d065517eae\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") "
Oct 30 19:23:38.654310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654215    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\" (UniqueName: \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\") pod \"bc364195-6c9a-4902-89a5-b2150d1f68b7\" (UID: \"bc364195-6c9a-4902-89a5-b2150d1f68b7\") "
Oct 30 19:23:38.654310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654306    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\" (UniqueName: \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\") pod \"893e6079-0241-4e01-a668-47d065517eae\" (UID: \"893e6079-0241-4e01-a668-47d065517eae\") "
Oct 30 19:23:38.654509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654469    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume" pod "bc364195-6c9a-4902-89a5-b2150d1f68b7" (UID: "bc364195-6c9a-4902-89a5-b2150d1f68b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:38.654967 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654939    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc364195-6c9a-4902-89a5-b2150d1f68b7-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.655111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.654977    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume" pod "893e6079-0241-4e01-a668-47d065517eae" (UID: "893e6079-0241-4e01-a668-47d065517eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:38.657360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.657338    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz" pod "bc364195-6c9a-4902-89a5-b2150d1f68b7" (UID: "bc364195-6c9a-4902-89a5-b2150d1f68b7"). InnerVolumeSpecName "kube-api-access-5gzpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:38.657529 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.657506    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m" pod "893e6079-0241-4e01-a668-47d065517eae" (UID: "893e6079-0241-4e01-a668-47d065517eae"). InnerVolumeSpecName "kube-api-access-rxw6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:38.713003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.708474    5065 generic.go:356] "Generic (PLEG): container finished" podID="893e6079-0241-4e01-a668-47d065517eae" containerID="9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d" exitCode=0
Oct 30 19:23:38.713003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.708533    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"
Oct 30 19:23:38.713003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.708564    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" event={"ID":"893e6079-0241-4e01-a668-47d065517eae","Type":"ContainerDied","Data":"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"}
Oct 30 19:23:38.713003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.708630    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx" event={"ID":"893e6079-0241-4e01-a668-47d065517eae","Type":"ContainerDied","Data":"6a09641bf4e5d0a3d141d5a10ea4fe1b7b6697eb08bdc93eba83324f11f6c2bb"}
Oct 30 19:23:38.713003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.708664    5065 scope.go:122] "RemoveContainer" containerID="9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"
Oct 30 19:23:38.713554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.713023    5065 generic.go:356] "Generic (PLEG): container finished" podID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerID="699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844" exitCode=0
Oct 30 19:23:38.713554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.713131    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" event={"ID":"bc364195-6c9a-4902-89a5-b2150d1f68b7","Type":"ContainerDied","Data":"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"}
Oct 30 19:23:38.713554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.713175    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w" event={"ID":"bc364195-6c9a-4902-89a5-b2150d1f68b7","Type":"ContainerDied","Data":"80bad14d34d29b7f12553bb6ab301fb930269d461789de584642d3d89dd0f9fd"}
Oct 30 19:23:38.713554 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.713248    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"
Oct 30 19:23:38.728123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.728072    5065 scope.go:122] "RemoveContainer" containerID="9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"
Oct 30 19:23:38.728383 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:38.728362    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d\": container with ID starting with 9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d not found: ID does not exist" containerID="9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"
Oct 30 19:23:38.728477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.728395    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d"} err="failed to get container status \"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d\": rpc error: code = NotFound desc = could not find container \"9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d\": container with ID starting with 9b3071e34ad17bdfd6d384cc20d7a9fde5b4e927173dd0d19fec2bcc9633bf3d not found: ID does not exist"
Oct 30 19:23:38.728477 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.728415    5065 scope.go:122] "RemoveContainer" containerID="699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"
Oct 30 19:23:38.740751 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.740736    5065 scope.go:122] "RemoveContainer" containerID="699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"
Oct 30 19:23:38.741060 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:38.741039    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844\": container with ID starting with 699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844 not found: ID does not exist" containerID="699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"
Oct 30 19:23:38.741179 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.741066    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844"} err="failed to get container status \"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844\": rpc error: code = NotFound desc = could not find container \"699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844\": container with ID starting with 699bbb3c1cd9ec9c3344102256576d1def83b8b494cd143bf16cbbda9a9cb844 not found: ID does not exist"
Oct 30 19:23:38.746370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.746322    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 19:23:38.752680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.752626    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ab7df0-7697b986f6-6pdrx"]
Oct 30 19:23:38.756963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.756938    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893e6079-0241-4e01-a668-47d065517eae-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.757051 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.756970    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzpz\" (UniqueName: \"kubernetes.io/projected/bc364195-6c9a-4902-89a5-b2150d1f68b7-kube-api-access-5gzpz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.757051 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.756988    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxw6m\" (UniqueName: \"kubernetes.io/projected/893e6079-0241-4e01-a668-47d065517eae-kube-api-access-rxw6m\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:38.772708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.772675    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 19:23:38.779495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:38.779467    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ab7df0-7c8548f45f-dwl6w"]
Oct 30 19:23:39.080153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:39.061218    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"]
Oct 30 19:23:39.080153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:39.079308    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035f83c-w7ks6"]
Oct 30 19:23:39.890239 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:39.890208    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893e6079-0241-4e01-a668-47d065517eae" path="/var/lib/kubelet/pods/893e6079-0241-4e01-a668-47d065517eae/volumes"
Oct 30 19:23:39.890841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:39.890774    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" path="/var/lib/kubelet/pods/bc364195-6c9a-4902-89a5-b2150d1f68b7/volumes"
Oct 30 19:23:39.892577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:39.891360    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da687236-bdfb-4726-9ca5-cedc3962790e" path="/var/lib/kubelet/pods/da687236-bdfb-4726-9ca5-cedc3962790e/volumes"
Oct 30 19:23:40.026175 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:40.026142    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"]
Oct 30 19:23:40.029590 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:40.029563    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ccd0c0-rdjgm"]
Oct 30 19:23:41.889979 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:41.889948    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1712b45c-892e-4608-b122-ea8202e3f191" path="/var/lib/kubelet/pods/1712b45c-892e-4608-b122-ea8202e3f191/volumes"
Oct 30 19:23:44.594300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.594268    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"]
Oct 30 19:23:44.595075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595054    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ffffacd1-2acc-4718-b3a0-79e09638f689" containerName="revoke-keys"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595076    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffffacd1-2acc-4718-b3a0-79e09638f689" containerName="revoke-keys"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595111    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595119    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595173    5065 cpu_manager.go:414] "containerMap: removing container" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595181    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing"
Oct 30 19:23:44.595197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595197    5065 cpu_manager.go:414] "containerMap: removing container" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595205    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595346    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="893e6079-0241-4e01-a668-47d065517eae" containerName="payload-processing"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595360    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc364195-6c9a-4902-89a5-b2150d1f68b7" containerName="payload-pre-processing"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595370    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595381    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffffacd1-2acc-4718-b3a0-79e09638f689" containerName="revoke-keys"
Oct 30 19:23:44.595422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.595389    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.600825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.600799    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.610061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.607455    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"]
Oct 30 19:23:44.610061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.608203    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.610061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.608220    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80abb5f-43ce-4af7-956a-54165637a734" containerName="istio-proxy"
Oct 30 19:23:44.610341 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.610150    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:44.629504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.612668    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"]
Oct 30 19:23:44.629504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.612765    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.629504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.612986    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:23:44.629504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.620972    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"]
Oct 30 19:23:44.747480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747448    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.747655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747500    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkvl\" (UniqueName: \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.747655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747524    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.747763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747669    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq5f5\" (UniqueName: \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.747763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747716    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.747886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.747859    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.849369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849298    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.849369 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849360    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.849589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849392    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkvl\" (UniqueName: \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.849589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849413    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.849589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849450    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5f5\" (UniqueName: \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.849589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.849494    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.850102 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.850059    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.850222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.850117    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.852101 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.852066    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.852173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.852072    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.863123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.863099    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5f5\" (UniqueName: \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\") pod \"maas-api-revoke-sub-fa13959e-z2nzq\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.863638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.863619    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkvl\" (UniqueName: \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\") pod \"maas-api-revoke-sub-a292131d-4wzdl\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:44.935585 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.935566    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:44.955289 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:44.955250    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:45.277133 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.277102    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"]
Oct 30 19:23:45.277656 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:23:45.277634    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeacfd19_775d_43e6_ba57_22d4e81991e2.slice/crio-39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8 WatchSource:0}: Error finding container 39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8: Status 404 returned error can't find the container with id 39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8
Oct 30 19:23:45.279296 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:23:45.279278    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d7d60_421f_4d8e_b4ad_7415c278d390.slice/crio-8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132 WatchSource:0}: Error finding container 8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132: Status 404 returned error can't find the container with id 8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132
Oct 30 19:23:45.281594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.281575    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"]
Oct 30 19:23:45.784851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.784805    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl" event={"ID":"371d7d60-421f-4d8e-b4ad-7415c278d390","Type":"ContainerStarted","Data":"dde50a6e5eb3b3d9e411ffe72af80b6b7cf44f3f6dedcef0668ce65a1f38f1b7"}
Oct 30 19:23:45.785269 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.784878    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl" event={"ID":"371d7d60-421f-4d8e-b4ad-7415c278d390","Type":"ContainerStarted","Data":"8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132"}
Oct 30 19:23:45.786845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.786819    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq" event={"ID":"beacfd19-775d-43e6-ba57-22d4e81991e2","Type":"ContainerStarted","Data":"86af5b20659d6ea08aaf806e84a69208e891c0780075e2680adc50539c8d0803"}
Oct 30 19:23:45.786964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.786867    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq" event={"ID":"beacfd19-775d-43e6-ba57-22d4e81991e2","Type":"ContainerStarted","Data":"39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8"}
Oct 30 19:23:45.855120 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.854995    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq" podStartSLOduration=1.854976557 podStartE2EDuration="1.854976557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:45.851507693 +0000 UTC m=+3046.114972641" watchObservedRunningTime="2026-10-30 19:23:45.854976557 +0000 UTC m=+3046.118441507"
Oct 30 19:23:45.855884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:45.855844    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl" podStartSLOduration=1.8558297179999999 podStartE2EDuration="1.855829718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:45.829236953 +0000 UTC m=+3046.092701926" watchObservedRunningTime="2026-10-30 19:23:45.855829718 +0000 UTC m=+3046.119294669"
Oct 30 19:23:46.814055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:46.813998    5065 generic.go:356] "Generic (PLEG): container finished" podID="371d7d60-421f-4d8e-b4ad-7415c278d390" containerID="dde50a6e5eb3b3d9e411ffe72af80b6b7cf44f3f6dedcef0668ce65a1f38f1b7" exitCode=0
Oct 30 19:23:46.814989 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:46.814963    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl" event={"ID":"371d7d60-421f-4d8e-b4ad-7415c278d390","Type":"ContainerDied","Data":"dde50a6e5eb3b3d9e411ffe72af80b6b7cf44f3f6dedcef0668ce65a1f38f1b7"}
Oct 30 19:23:46.818140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:46.818108    5065 generic.go:356] "Generic (PLEG): container finished" podID="beacfd19-775d-43e6-ba57-22d4e81991e2" containerID="86af5b20659d6ea08aaf806e84a69208e891c0780075e2680adc50539c8d0803" exitCode=0
Oct 30 19:23:46.818273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:46.818196    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq" event={"ID":"beacfd19-775d-43e6-ba57-22d4e81991e2","Type":"ContainerDied","Data":"86af5b20659d6ea08aaf806e84a69208e891c0780075e2680adc50539c8d0803"}
Oct 30 19:23:48.054941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.054912    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:48.089974 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.089913    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:48.202555 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202511    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\" (UniqueName: \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\") pod \"beacfd19-775d-43e6-ba57-22d4e81991e2\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") "
Oct 30 19:23:48.202729 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202684    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\") pod \"371d7d60-421f-4d8e-b4ad-7415c278d390\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") "
Oct 30 19:23:48.202810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202794    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\") pod \"371d7d60-421f-4d8e-b4ad-7415c278d390\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") "
Oct 30 19:23:48.202893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202843    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\" (UniqueName: \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\") pod \"371d7d60-421f-4d8e-b4ad-7415c278d390\" (UID: \"371d7d60-421f-4d8e-b4ad-7415c278d390\") "
Oct 30 19:23:48.202960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202914    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\") pod \"beacfd19-775d-43e6-ba57-22d4e81991e2\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") "
Oct 30 19:23:48.202960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.202945    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\") pod \"beacfd19-775d-43e6-ba57-22d4e81991e2\" (UID: \"beacfd19-775d-43e6-ba57-22d4e81991e2\") "
Oct 30 19:23:48.203649 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.203531    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca" pod "beacfd19-775d-43e6-ba57-22d4e81991e2" (UID: "beacfd19-775d-43e6-ba57-22d4e81991e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:48.203886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.203814    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca" pod "371d7d60-421f-4d8e-b4ad-7415c278d390" (UID: "371d7d60-421f-4d8e-b4ad-7415c278d390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:48.205228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.205183    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca" pod "beacfd19-775d-43e6-ba57-22d4e81991e2" (UID: "beacfd19-775d-43e6-ba57-22d4e81991e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:48.205228 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.205196    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5" pod "beacfd19-775d-43e6-ba57-22d4e81991e2" (UID: "beacfd19-775d-43e6-ba57-22d4e81991e2"). InnerVolumeSpecName "kube-api-access-kq5f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:48.205480 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.205457    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca" pod "371d7d60-421f-4d8e-b4ad-7415c278d390" (UID: "371d7d60-421f-4d8e-b4ad-7415c278d390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:48.205608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.205559    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl" pod "371d7d60-421f-4d8e-b4ad-7415c278d390" (UID: "371d7d60-421f-4d8e-b4ad-7415c278d390"). InnerVolumeSpecName "kube-api-access-kwkvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:48.304166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304141    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.304276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304168    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beacfd19-775d-43e6-ba57-22d4e81991e2-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.304276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304178    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq5f5\" (UniqueName: \"kubernetes.io/projected/beacfd19-775d-43e6-ba57-22d4e81991e2-kube-api-access-kq5f5\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.304276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304187    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.304276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304196    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371d7d60-421f-4d8e-b4ad-7415c278d390-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.304276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.304206    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkvl\" (UniqueName: \"kubernetes.io/projected/371d7d60-421f-4d8e-b4ad-7415c278d390-kube-api-access-kwkvl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:48.836677 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.836647    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"
Oct 30 19:23:48.836907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.836686    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl" event={"ID":"371d7d60-421f-4d8e-b4ad-7415c278d390","Type":"ContainerDied","Data":"8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132"}
Oct 30 19:23:48.836907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.836731    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bfba2f68954ca2f9cdb0f1a0b3df2a0ec452b46e459a89177d5ef90473fa132"
Oct 30 19:23:48.838886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.838855    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"
Oct 30 19:23:48.839018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.838675    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq" event={"ID":"beacfd19-775d-43e6-ba57-22d4e81991e2","Type":"ContainerDied","Data":"39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8"}
Oct 30 19:23:48.839018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:48.839011    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39cd8723275019e5fc75d3bcbcfc48a5423d29d8e36f8e6963ca5c260ebec5b8"
Oct 30 19:23:52.084108 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:52.078819    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"]
Oct 30 19:23:52.105467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:52.085509    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3028cb3-f8zk6"]
Oct 30 19:23:53.059573 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:23:53.059535    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:23:53.936794 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:53.934403    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982b884d-1b16-4757-a607-6545820d6d10" path="/var/lib/kubelet/pods/982b884d-1b16-4757-a607-6545820d6d10/volumes"
Oct 30 19:23:58.352047 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:58.351972    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"]
Oct 30 19:23:58.353405 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:58.353372    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7a8fa2-sdn2j"]
Oct 30 19:23:59.891154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:23:59.891126    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffffacd1-2acc-4718-b3a0-79e09638f689" path="/var/lib/kubelet/pods/ffffacd1-2acc-4718-b3a0-79e09638f689/volumes"
Oct 30 19:24:06.701550 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:06.701519    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 19:24:06.702003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:06.701727    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy" containerID="cri-o://0bf23cc51402ccf8582c06caec8b4414273b51d859f9ed3f1b25753d99d85521" gracePeriod=30
Oct 30 19:24:06.705185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:06.705156    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 19:24:06.705373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:06.705353    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy" containerID="cri-o://37a46763469e5239bab9688fe59d8bafc109a599d416c093966f315cdf93ea32" gracePeriod=30
Oct 30 19:24:10.035761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:10.035724    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"]
Oct 30 19:24:10.039042 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:10.039018    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05931ac2-kcdsp"]
Oct 30 19:24:11.923149 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:11.923110    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28d7286-1061-4515-8568-e952ed62ca6c" path="/var/lib/kubelet/pods/d28d7286-1061-4515-8568-e952ed62ca6c/volumes"
Oct 30 19:24:12.087739 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.087585    5065 generic.go:356] "Generic (PLEG): container finished" podID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerID="37a46763469e5239bab9688fe59d8bafc109a599d416c093966f315cdf93ea32" exitCode=0
Oct 30 19:24:12.088049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.087665    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" event={"ID":"c88fb979-a4e3-4b70-92eb-a2030d0540f6","Type":"ContainerDied","Data":"37a46763469e5239bab9688fe59d8bafc109a599d416c093966f315cdf93ea32"}
Oct 30 19:24:12.090630 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.090602    5065 generic.go:356] "Generic (PLEG): container finished" podID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerID="0bf23cc51402ccf8582c06caec8b4414273b51d859f9ed3f1b25753d99d85521" exitCode=0
Oct 30 19:24:12.090886 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.090732    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" event={"ID":"ebd75150-bfd0-40c6-ab71-9cb0e107cee8","Type":"ContainerDied","Data":"0bf23cc51402ccf8582c06caec8b4414273b51d859f9ed3f1b25753d99d85521"}
Oct 30 19:24:12.090999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.090924    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr" event={"ID":"ebd75150-bfd0-40c6-ab71-9cb0e107cee8","Type":"ContainerDied","Data":"da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443"}
Oct 30 19:24:12.090999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.090956    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4adf7d1d4c353d26f1f95128e65858ea26709a41757ec1053c665be8762443"
Oct 30 19:24:12.094254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.094236    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 19:24:12.161872 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.161846    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 19:24:12.171660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171637    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.171797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171700    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.171797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171748    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.171924 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171795    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.171994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171923    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.171994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.171960    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.172196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172134    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.172275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172211    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.172379 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172359    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.172421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172379    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:12.172421 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172043    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.172516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172489    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.172571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172523    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\") pod \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\" (UID: \"ebd75150-bfd0-40c6-ab71-9cb0e107cee8\") "
Oct 30 19:24:12.172764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.172743    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.173221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.173194    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.173325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.173222    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.173325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.173236    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.173325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.173249    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.173325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.173263    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.174418 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.174394    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.174663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.174647    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "kube-api-access-z2s85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.174758 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.174661    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.174825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.174794    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo" pod "ebd75150-bfd0-40c6-ab71-9cb0e107cee8" (UID: "ebd75150-bfd0-40c6-ab71-9cb0e107cee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:12.274031 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274004    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274131    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274202    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274247 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274246    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274267    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274341    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.274436 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274409    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274466    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274527    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.274631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274547    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274631 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274624    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\") pod \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\" (UID: \"c88fb979-a4e3-4b70-92eb-a2030d0540f6\") "
Oct 30 19:24:12.274868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274641    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.274868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274816    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.274868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.274846    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:12.275225 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275200    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275228    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275251    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275267    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275281    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275296    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2s85\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-kube-api-access-z2s85\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275309    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd75150-bfd0-40c6-ab71-9cb0e107cee8-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275323    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.275342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.275337    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.277117 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.277073    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:12.277266 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.277250    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "kube-api-access-smn7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.277415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.277395    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.277481 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.277434    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token" pod "c88fb979-a4e3-4b70-92eb-a2030d0540f6" (UID: "c88fb979-a4e3-4b70-92eb-a2030d0540f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.376896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.376833    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.376896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.376859    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smn7z\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-kube-api-access-smn7z\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.376896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.376869    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.376896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:12.376880    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88fb979-a4e3-4b70-92eb-a2030d0540f6-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:13.101484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.101453    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"
Oct 30 19:24:13.101991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.101495    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z" event={"ID":"c88fb979-a4e3-4b70-92eb-a2030d0540f6","Type":"ContainerDied","Data":"ecd0c156fb3113f840c8d10b31178d97f9a3c9c9bd08ac431ff431f2c241893d"}
Oct 30 19:24:13.101991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.101566    5065 scope.go:122] "RemoveContainer" containerID="37a46763469e5239bab9688fe59d8bafc109a599d416c093966f315cdf93ea32"
Oct 30 19:24:13.101991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.101622    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"
Oct 30 19:24:13.133339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.133314    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 19:24:13.138541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.138518    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-v7fkr"]
Oct 30 19:24:13.151609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.151580    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 19:24:13.156508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.156489    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-bg22z"]
Oct 30 19:24:13.889482 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.889443    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" path="/var/lib/kubelet/pods/c88fb979-a4e3-4b70-92eb-a2030d0540f6/volumes"
Oct 30 19:24:13.889968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:13.889954    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" path="/var/lib/kubelet/pods/ebd75150-bfd0-40c6-ab71-9cb0e107cee8/volumes"
Oct 30 19:24:16.407049 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:16.407012    5065 scope.go:122] "RemoveContainer" containerID="adec58811b6f325c32039c84b53b008cb7e0afc23e49044bcf36f92c4a861619"
Oct 30 19:24:16.424544 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:16.424520    5065 scope.go:122] "RemoveContainer" containerID="0bf23cc51402ccf8582c06caec8b4414273b51d859f9ed3f1b25753d99d85521"
Oct 30 19:24:20.026783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:20.026744    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"]
Oct 30 19:24:20.029898 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:20.029872    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f04cb2-tdq9d"]
Oct 30 19:24:21.906044 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:21.906008    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c0c782-5018-45e2-adf9-b71d3b7563fc" path="/var/lib/kubelet/pods/40c0c782-5018-45e2-adf9-b71d3b7563fc/volumes"
Oct 30 19:24:23.057351 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:24:23.057316    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:24:31.027032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:31.026994    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"]
Oct 30 19:24:31.030537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:31.030508    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288c99c4-dhb7v"]
Oct 30 19:24:31.889373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:31.889336    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce5c531-d4c4-4ef7-a317-3014d15e4a3a" path="/var/lib/kubelet/pods/9ce5c531-d4c4-4ef7-a317-3014d15e4a3a/volumes"
Oct 30 19:24:35.747859 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:35.747817    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:24:35.748413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:35.748063    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery" containerID="cri-o://01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53" gracePeriod=30
Oct 30 19:24:36.022041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.022006    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:24:36.068229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068197    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068290    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068338    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068404 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068374    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068427    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068604 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068472    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.068732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.068631    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\") pod \"fbec99a3-129b-4049-92ff-ac6f73c69e54\" (UID: \"fbec99a3-129b-4049-92ff-ac6f73c69e54\") "
Oct 30 19:24:36.069291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.069244    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:36.071999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.071969    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.072310 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.072278    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:36.072443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.072414    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.072443 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.072429    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.072576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.072548    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:36.072993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.072964    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg" pod "fbec99a3-129b-4049-92ff-ac6f73c69e54" (UID: "fbec99a3-129b-4049-92ff-ac6f73c69e54"). InnerVolumeSpecName "kube-api-access-hrhpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:36.170136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170064    5065 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-dns-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170128    5065 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-cacerts\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170136 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170139    5065 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-kubeconfig\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170150    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhpg\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-kube-api-access-hrhpg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170160    5065 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-csr-ca-configmap\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170168    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbec99a3-129b-4049-92ff-ac6f73c69e54-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.170387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.170177    5065 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbec99a3-129b-4049-92ff-ac6f73c69e54-local-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.303003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.302963    5065 generic.go:356] "Generic (PLEG): container finished" podID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerID="01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53" exitCode=0
Oct 30 19:24:36.303275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.303017    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"
Oct 30 19:24:36.303275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.303047    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" event={"ID":"fbec99a3-129b-4049-92ff-ac6f73c69e54","Type":"ContainerDied","Data":"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"}
Oct 30 19:24:36.303275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.303147    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp" event={"ID":"fbec99a3-129b-4049-92ff-ac6f73c69e54","Type":"ContainerDied","Data":"31cb6bfd93fb486d8776f852e6eb69693b3206dc5cbdf5287444bad8f9ec549f"}
Oct 30 19:24:36.303275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.303184    5065 scope.go:122] "RemoveContainer" containerID="01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"
Oct 30 19:24:36.316363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.316344    5065 scope.go:122] "RemoveContainer" containerID="01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"
Oct 30 19:24:36.316622 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:24:36.316599    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53\": container with ID starting with 01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53 not found: ID does not exist" containerID="01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"
Oct 30 19:24:36.316704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.316635    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53"} err="failed to get container status \"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53\": rpc error: code = NotFound desc = could not find container \"01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53\": container with ID starting with 01177bd3d2f54385e7bc976c66c45aaf9142d956e53032ea0389e23d0f81de53 not found: ID does not exist"
Oct 30 19:24:36.331473 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.331450    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:24:36.334626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:36.334607    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jm7mp"]
Oct 30 19:24:37.890594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:37.890556    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" path="/var/lib/kubelet/pods/fbec99a3-129b-4049-92ff-ac6f73c69e54/volumes"
Oct 30 19:24:42.027456 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:42.027359    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"]
Oct 30 19:24:42.030499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:42.030476    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-916dc787-ncqnx"]
Oct 30 19:24:43.889940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:24:43.889906    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953323c3-3705-45d5-8e2e-0d1b6716023b" path="/var/lib/kubelet/pods/953323c3-3705-45d5-8e2e-0d1b6716023b/volumes"
Oct 30 19:24:53.056604 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:24:53.056567    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:25:03.337241 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337195    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"]
Oct 30 19:25:03.337791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337744    5065 cpu_manager.go:414] "containerMap: removing container" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery"
Oct 30 19:25:03.337791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337755    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery"
Oct 30 19:25:03.337791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337768    5065 cpu_manager.go:414] "containerMap: removing container" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy"
Oct 30 19:25:03.337791 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337773    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337795    5065 cpu_manager.go:414] "containerMap: removing container" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337800    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337814    5065 cpu_manager.go:414] "containerMap: removing container" podUID="371d7d60-421f-4d8e-b4ad-7415c278d390" containerName="revoke-keys"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337819    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371d7d60-421f-4d8e-b4ad-7415c278d390" containerName="revoke-keys"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337826    5065 cpu_manager.go:414] "containerMap: removing container" podUID="beacfd19-775d-43e6-ba57-22d4e81991e2" containerName="revoke-keys"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337831    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beacfd19-775d-43e6-ba57-22d4e81991e2" containerName="revoke-keys"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337907    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbec99a3-129b-4049-92ff-ac6f73c69e54" containerName="discovery"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337914    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88fb979-a4e3-4b70-92eb-a2030d0540f6" containerName="istio-proxy"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337923    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd75150-bfd0-40c6-ab71-9cb0e107cee8" containerName="istio-proxy"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337931    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="371d7d60-421f-4d8e-b4ad-7415c278d390" containerName="revoke-keys"
Oct 30 19:25:03.337940 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.337938    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="beacfd19-775d-43e6-ba57-22d4e81991e2" containerName="revoke-keys"
Oct 30 19:25:03.341320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.341298    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.348471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.348436    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:25:03.348639 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.348507    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:03.353400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.353372    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"]
Oct 30 19:25:03.385099 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.385061    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.385325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.385193    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.385325 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.385221    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhh6\" (UniqueName: \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.486761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.486724    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.486964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.486771    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhh6\" (UniqueName: \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.486964 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.486880    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.487571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.487546    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.489640 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.489613    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.501067 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.501036    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhh6\" (UniqueName: \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\") pod \"maas-api-revoke-sub-174ee13d-nkxqt\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.658884 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.658783    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:03.955609 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:03.955565    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"]
Oct 30 19:25:04.529720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:04.529680    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt" event={"ID":"af31fba6-39b1-4c2f-81d3-344cbea4ebf9","Type":"ContainerStarted","Data":"4a0a314a6df9fa2e2094bd5f5f94e9d2f18c4b745a5934b752c59f3dd2be0773"}
Oct 30 19:25:04.530173 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:04.529739    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt" event={"ID":"af31fba6-39b1-4c2f-81d3-344cbea4ebf9","Type":"ContainerStarted","Data":"a81fbee28ebc8006ee2e538e4ea4923f11e366cafea01145f14b26bb10da07bc"}
Oct 30 19:25:04.551589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:04.551529    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt" podStartSLOduration=1.551516714 podStartE2EDuration="1.551516714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:04.550446847 +0000 UTC m=+3124.813911815" watchObservedRunningTime="2026-10-30 19:25:04.551516714 +0000 UTC m=+3124.814981662"
Oct 30 19:25:05.539484 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:05.539448    5065 generic.go:356] "Generic (PLEG): container finished" podID="af31fba6-39b1-4c2f-81d3-344cbea4ebf9" containerID="4a0a314a6df9fa2e2094bd5f5f94e9d2f18c4b745a5934b752c59f3dd2be0773" exitCode=0
Oct 30 19:25:05.540004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:05.539538    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt" event={"ID":"af31fba6-39b1-4c2f-81d3-344cbea4ebf9","Type":"ContainerDied","Data":"4a0a314a6df9fa2e2094bd5f5f94e9d2f18c4b745a5934b752c59f3dd2be0773"}
Oct 30 19:25:06.715975 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.715944    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:06.828040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.827951    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\") pod \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") "
Oct 30 19:25:06.828294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.828271    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\") pod \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") "
Oct 30 19:25:06.828345 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.828328    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\" (UniqueName: \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\") pod \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\" (UID: \"af31fba6-39b1-4c2f-81d3-344cbea4ebf9\") "
Oct 30 19:25:06.828439 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.828419    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca" pod "af31fba6-39b1-4c2f-81d3-344cbea4ebf9" (UID: "af31fba6-39b1-4c2f-81d3-344cbea4ebf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:06.828686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.828656    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:06.830683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.830653    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca" pod "af31fba6-39b1-4c2f-81d3-344cbea4ebf9" (UID: "af31fba6-39b1-4c2f-81d3-344cbea4ebf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:06.830808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.830679    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6" pod "af31fba6-39b1-4c2f-81d3-344cbea4ebf9" (UID: "af31fba6-39b1-4c2f-81d3-344cbea4ebf9"). InnerVolumeSpecName "kube-api-access-6dhh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:06.930267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.930214    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:06.930267 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:06.930269    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhh6\" (UniqueName: \"kubernetes.io/projected/af31fba6-39b1-4c2f-81d3-344cbea4ebf9-kube-api-access-6dhh6\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:07.558311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:07.558268    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt" event={"ID":"af31fba6-39b1-4c2f-81d3-344cbea4ebf9","Type":"ContainerDied","Data":"a81fbee28ebc8006ee2e538e4ea4923f11e366cafea01145f14b26bb10da07bc"}
Oct 30 19:25:07.558516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:07.558325    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81fbee28ebc8006ee2e538e4ea4923f11e366cafea01145f14b26bb10da07bc"
Oct 30 19:25:07.558516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:07.558321    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174ee13d-nkxqt"
Oct 30 19:25:11.551619 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:11.551582    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:25:11.552132 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:11.551816    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy" containerID="cri-o://a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd" gracePeriod=30
Oct 30 19:25:11.555074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:11.555048    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:25:11.555296 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:11.555261    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy" containerID="cri-o://14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90" gracePeriod=30
Oct 30 19:25:14.462096 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.462050    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"]
Oct 30 19:25:14.462602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.462586    5065 cpu_manager.go:414] "containerMap: removing container" podUID="af31fba6-39b1-4c2f-81d3-344cbea4ebf9" containerName="revoke-keys"
Oct 30 19:25:14.462646 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.462603    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af31fba6-39b1-4c2f-81d3-344cbea4ebf9" containerName="revoke-keys"
Oct 30 19:25:14.462700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.462691    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="af31fba6-39b1-4c2f-81d3-344cbea4ebf9" containerName="revoke-keys"
Oct 30 19:25:14.466638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.466620    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.472985 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.472958    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:14.474749 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.474681    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:25:14.481633 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.481615    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"]
Oct 30 19:25:14.521363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.521328    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.521363 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.521366    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxsb\" (UniqueName: \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.521567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.521502    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.623201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.623167    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.623385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.623306    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.623385 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.623326    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxsb\" (UniqueName: \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.623804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.623784    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.625852 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.625823    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.639415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.639390    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxsb\" (UniqueName: \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\") pod \"maas-api-revoke-sub-5ee29955-l4nz9\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:14.783415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:14.783379    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:15.087197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:15.087175    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"]
Oct 30 19:25:15.090074 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:25:15.090040    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b134a8_8eef_4dc2_a0e0_3ca5b0c9ed2e.slice/crio-08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f WatchSource:0}: Error finding container 08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f: Status 404 returned error can't find the container with id 08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f
Oct 30 19:25:15.624911 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:15.624876    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9" event={"ID":"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e","Type":"ContainerStarted","Data":"36d9d31e0eec2d5bc9f24f7a848820277e260f8b6b5e07d3278a008923226171"}
Oct 30 19:25:15.625429 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:15.624939    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9" event={"ID":"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e","Type":"ContainerStarted","Data":"08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f"}
Oct 30 19:25:15.662140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:15.659616    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9" podStartSLOduration=1.659589355 podStartE2EDuration="1.659589355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:15.647629052 +0000 UTC m=+3135.911094012" watchObservedRunningTime="2026-10-30 19:25:15.659589355 +0000 UTC m=+3135.923054304"
Oct 30 19:25:16.485320 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.485239    5065 scope.go:122] "RemoveContainer" containerID="2ade67e4638b2c036527090f8a34c6b61acc2869c5b721de422deb9a42fcc2e5"
Oct 30 19:25:16.498813 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.498782    5065 scope.go:122] "RemoveContainer" containerID="070794516a27ad067b385b84204bc8eab83020d962989a44617723602d84a80b"
Oct 30 19:25:16.512207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.512182    5065 scope.go:122] "RemoveContainer" containerID="e7e151d663d52a1a572e4d41c49462c82754a52b223403db9747b8ffa2ccf22d"
Oct 30 19:25:16.524541 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.524509    5065 scope.go:122] "RemoveContainer" containerID="8004e381ce076ea4d139f91439902ee46dbe050be644412630457f7b935caa10"
Oct 30 19:25:16.534812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.534773    5065 scope.go:122] "RemoveContainer" containerID="1eff6989007d7bda687fc1fdafb7c707dd0ff412ee95c1995bc6650eb44c89bb"
Oct 30 19:25:16.543838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.543818    5065 scope.go:122] "RemoveContainer" containerID="61f72ab380b8665d37b539d09117e4047062e10520350fb6e4627bfb397da808"
Oct 30 19:25:16.552812 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.552796    5065 scope.go:122] "RemoveContainer" containerID="6b827a5094445dba5c4621d00017b4231878732e4b87dfcc45a87f190b87ffb0"
Oct 30 19:25:16.645811 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.645777    5065 generic.go:356] "Generic (PLEG): container finished" podID="98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" containerID="36d9d31e0eec2d5bc9f24f7a848820277e260f8b6b5e07d3278a008923226171" exitCode=0
Oct 30 19:25:16.646220 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.645901    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9" event={"ID":"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e","Type":"ContainerDied","Data":"36d9d31e0eec2d5bc9f24f7a848820277e260f8b6b5e07d3278a008923226171"}
Oct 30 19:25:16.863075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.863052    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:25:16.880866 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.880846    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:25:16.954378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954348    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.954542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954402    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.954542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954455    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.954542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954514    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.954722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954612    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.954722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954648    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.954722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954658    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.954722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954690    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.954722 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954718    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954747    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954770    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954786    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954792    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954847    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954907    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954928    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954937    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954946    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955000 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.954968    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.955645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.955040    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.955645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.955234    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:16.955645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.955430    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\") pod \"e497fb69-8848-428d-ade7-735218c8637c\" (UID: \"e497fb69-8848-428d-ade7-735218c8637c\") "
Oct 30 19:25:16.955645 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.955464    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\") pod \"8035a145-4645-400e-8237-79e4def5577e\" (UID: \"8035a145-4645-400e-8237-79e4def5577e\") "
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956100    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956240    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035a145-4645-400e-8237-79e4def5577e-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956262    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956278    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e497fb69-8848-428d-ade7-735218c8637c-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956361    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956393    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956397    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956429    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956448    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956645    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.956982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.956712    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.958203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.958161    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:16.958626 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.958558    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.958976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.958768    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "kube-api-access-hrr9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:16.958976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.958867    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:16.958976 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.958939    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token" pod "8035a145-4645-400e-8237-79e4def5577e" (UID: "8035a145-4645-400e-8237-79e4def5577e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:16.959230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.959210    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:16.959787 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.959767    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:16.960156 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:16.960142    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl" pod "e497fb69-8848-428d-ade7-735218c8637c" (UID: "e497fb69-8848-428d-ade7-735218c8637c"). InnerVolumeSpecName "kube-api-access-qzcfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:17.057356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057319    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e497fb69-8848-428d-ade7-735218c8637c-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057354    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057364    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035a145-4645-400e-8237-79e4def5577e-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057374    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrr9k\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-kube-api-access-hrr9k\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057385    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057395    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057406    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035a145-4645-400e-8237-79e4def5577e-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057415    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057429    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057439    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcfl\" (UniqueName: \"kubernetes.io/projected/e497fb69-8848-428d-ade7-735218c8637c-kube-api-access-qzcfl\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057448    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e497fb69-8848-428d-ade7-735218c8637c-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.057602 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.057456    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035a145-4645-400e-8237-79e4def5577e-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.659010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.658970    5065 generic.go:356] "Generic (PLEG): container finished" podID="8035a145-4645-400e-8237-79e4def5577e" containerID="14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90" exitCode=0
Oct 30 19:25:17.659568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.659031    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"
Oct 30 19:25:17.659568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.659053    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" event={"ID":"8035a145-4645-400e-8237-79e4def5577e","Type":"ContainerDied","Data":"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"}
Oct 30 19:25:17.659568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.659127    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb" event={"ID":"8035a145-4645-400e-8237-79e4def5577e","Type":"ContainerDied","Data":"0fc30b3125a4d1db5e70dce5cf080b8b3554ab925aca2fc9fab758d2791bb808"}
Oct 30 19:25:17.659568 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.659161    5065 scope.go:122] "RemoveContainer" containerID="14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"
Oct 30 19:25:17.661739 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.661710    5065 generic.go:356] "Generic (PLEG): container finished" podID="e497fb69-8848-428d-ade7-735218c8637c" containerID="a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd" exitCode=0
Oct 30 19:25:17.662243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.661768    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"
Oct 30 19:25:17.662243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.661797    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" event={"ID":"e497fb69-8848-428d-ade7-735218c8637c","Type":"ContainerDied","Data":"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"}
Oct 30 19:25:17.662243 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.661855    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh" event={"ID":"e497fb69-8848-428d-ade7-735218c8637c","Type":"ContainerDied","Data":"42e83501e8526c0fff446eeab47b33626e6be52d234f14efce0493fd2ef3f84e"}
Oct 30 19:25:17.676076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.676055    5065 scope.go:122] "RemoveContainer" containerID="14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"
Oct 30 19:25:17.677566 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:25:17.677537    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90\": container with ID starting with 14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90 not found: ID does not exist" containerID="14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"
Oct 30 19:25:17.677682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.677604    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90"} err="failed to get container status \"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90\": rpc error: code = NotFound desc = could not find container \"14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90\": container with ID starting with 14df1e6fe8154124698385864c318bad6937ada28221c3ac2d6aa6915afcbc90 not found: ID does not exist"
Oct 30 19:25:17.677682 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.677631    5065 scope.go:122] "RemoveContainer" containerID="a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"
Oct 30 19:25:17.692172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.692154    5065 scope.go:122] "RemoveContainer" containerID="a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"
Oct 30 19:25:17.692565 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:25:17.692517    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd\": container with ID starting with a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd not found: ID does not exist" containerID="a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"
Oct 30 19:25:17.692669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.692577    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd"} err="failed to get container status \"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd\": rpc error: code = NotFound desc = could not find container \"a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd\": container with ID starting with a9eaa460470057cd18190909168221eb9f738ecbe7d3a237a30e9cb0df7363bd not found: ID does not exist"
Oct 30 19:25:17.719909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.719880    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:25:17.736843 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.736813    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-s68fh"]
Oct 30 19:25:17.818500 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.818472    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:25:17.822425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.822397    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-66lbb"]
Oct 30 19:25:17.828673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.828646    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:17.893735 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.893701    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8035a145-4645-400e-8237-79e4def5577e" path="/var/lib/kubelet/pods/8035a145-4645-400e-8237-79e4def5577e/volumes"
Oct 30 19:25:17.894663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.894640    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e497fb69-8848-428d-ade7-735218c8637c" path="/var/lib/kubelet/pods/e497fb69-8848-428d-ade7-735218c8637c/volumes"
Oct 30 19:25:17.984130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.984012    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\" (UniqueName: \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\") pod \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") "
Oct 30 19:25:17.984130 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.984109    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\") pod \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") "
Oct 30 19:25:17.984370 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.984197    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\") pod \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\" (UID: \"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e\") "
Oct 30 19:25:17.984801 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.984768    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca" pod "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" (UID: "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:17.984972 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.984915    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:17.986516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.986492    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb" pod "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" (UID: "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e"). InnerVolumeSpecName "kube-api-access-bzxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:17.986623 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:17.986609    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca" pod "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" (UID: "98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:18.086077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:18.086054    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzxsb\" (UniqueName: \"kubernetes.io/projected/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-kube-api-access-bzxsb\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.086077 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:18.086079    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.670663 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:18.670634    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9"
Oct 30 19:25:18.671111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:18.670696    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee29955-l4nz9" event={"ID":"98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e","Type":"ContainerDied","Data":"08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f"}
Oct 30 19:25:18.671111 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:18.670738    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b2781cf284c650df5a9bbab1dc1b756900d679f3e9ab0534e676de000e327f"
Oct 30 19:25:23.059538 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:25:23.059502    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:25:25.447003 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.446962    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"]
Oct 30 19:25:25.447517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447502    5065 cpu_manager.go:414] "containerMap: removing container" podUID="98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" containerName="revoke-keys"
Oct 30 19:25:25.447517 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447518    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" containerName="revoke-keys"
Oct 30 19:25:25.447669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447529    5065 cpu_manager.go:414] "containerMap: removing container" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy"
Oct 30 19:25:25.447669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447534    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy"
Oct 30 19:25:25.447669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447580    5065 cpu_manager.go:414] "containerMap: removing container" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy"
Oct 30 19:25:25.447669 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447586    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy"
Oct 30 19:25:25.447889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447690    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="8035a145-4645-400e-8237-79e4def5577e" containerName="istio-proxy"
Oct 30 19:25:25.447889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447710    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e" containerName="revoke-keys"
Oct 30 19:25:25.447889 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.447722    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="e497fb69-8848-428d-ade7-735218c8637c" containerName="istio-proxy"
Oct 30 19:25:25.452393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.452374    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.476168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.476141    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:25:25.491680 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.491661    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:25.499373 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.499355    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"]
Oct 30 19:25:25.569778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.569746    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2t49\" (UniqueName: \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.569934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.569791    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.569934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.569863    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.670994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.670967    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2t49\" (UniqueName: \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.671178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.671005    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.671178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.671031    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.671775 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.671747    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.673623 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.673602    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.688692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.688670    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2t49\" (UniqueName: \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\") pod \"maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:25.767402 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:25.767378    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:26.061935 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:26.061907    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"]
Oct 30 19:25:26.062290 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:25:26.062265    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f85727e_2a60_41dc_8000_9be33b6bfdaa.slice/crio-9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3 WatchSource:0}: Error finding container 9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3: Status 404 returned error can't find the container with id 9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3
Oct 30 19:25:26.745608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:26.745520    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww" event={"ID":"0f85727e-2a60-41dc-8000-9be33b6bfdaa","Type":"ContainerStarted","Data":"5b38c92f48a93a740a1cad054c7ea2581cad127f5e158386d5473b34898eb27b"}
Oct 30 19:25:26.745608 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:26.745576    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww" event={"ID":"0f85727e-2a60-41dc-8000-9be33b6bfdaa","Type":"ContainerStarted","Data":"9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3"}
Oct 30 19:25:26.769741 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:26.769698    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww" podStartSLOduration=1.769685111 podStartE2EDuration="1.769685111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:26.768521373 +0000 UTC m=+3147.031986339" watchObservedRunningTime="2026-10-30 19:25:26.769685111 +0000 UTC m=+3147.033150121"
Oct 30 19:25:27.755468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:27.755438    5065 generic.go:356] "Generic (PLEG): container finished" podID="0f85727e-2a60-41dc-8000-9be33b6bfdaa" containerID="5b38c92f48a93a740a1cad054c7ea2581cad127f5e158386d5473b34898eb27b" exitCode=0
Oct 30 19:25:27.755969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:27.755536    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww" event={"ID":"0f85727e-2a60-41dc-8000-9be33b6bfdaa","Type":"ContainerDied","Data":"5b38c92f48a93a740a1cad054c7ea2581cad127f5e158386d5473b34898eb27b"}
Oct 30 19:25:28.918452 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:28.918418    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:29.014165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.014134    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\" (UniqueName: \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\") pod \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") "
Oct 30 19:25:29.014359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.014185    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\") pod \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") "
Oct 30 19:25:29.014359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.014314    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\") pod \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\" (UID: \"0f85727e-2a60-41dc-8000-9be33b6bfdaa\") "
Oct 30 19:25:29.014600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.014573    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca" pod "0f85727e-2a60-41dc-8000-9be33b6bfdaa" (UID: "0f85727e-2a60-41dc-8000-9be33b6bfdaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:29.014810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.014790    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.016537 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.016511    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49" pod "0f85727e-2a60-41dc-8000-9be33b6bfdaa" (UID: "0f85727e-2a60-41dc-8000-9be33b6bfdaa"). InnerVolumeSpecName "kube-api-access-m2t49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:29.016951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.016930    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca" pod "0f85727e-2a60-41dc-8000-9be33b6bfdaa" (UID: "0f85727e-2a60-41dc-8000-9be33b6bfdaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:29.115892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.115859    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2t49\" (UniqueName: \"kubernetes.io/projected/0f85727e-2a60-41dc-8000-9be33b6bfdaa-kube-api-access-m2t49\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.115892 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.115890    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f85727e-2a60-41dc-8000-9be33b6bfdaa-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.776559 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.776533    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"
Oct 30 19:25:29.776822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.776533    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww" event={"ID":"0f85727e-2a60-41dc-8000-9be33b6bfdaa","Type":"ContainerDied","Data":"9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3"}
Oct 30 19:25:29.776951 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:29.776849    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9803f8983e974531de381a33876e65d6816b85cd482c6c6c23a2e6f67c8acda3"
Oct 30 19:25:37.513449 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:37.513406    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"]
Oct 30 19:25:37.516430 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:37.516407    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fe8ece-z7kww"]
Oct 30 19:25:37.889687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:37.889583    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f85727e-2a60-41dc-8000-9be33b6bfdaa" path="/var/lib/kubelet/pods/0f85727e-2a60-41dc-8000-9be33b6bfdaa/volumes"
Oct 30 19:25:46.561577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.561539    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"]
Oct 30 19:25:46.562030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.562014    5065 cpu_manager.go:414] "containerMap: removing container" podUID="0f85727e-2a60-41dc-8000-9be33b6bfdaa" containerName="revoke-keys"
Oct 30 19:25:46.562030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.562028    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f85727e-2a60-41dc-8000-9be33b6bfdaa" containerName="revoke-keys"
Oct 30 19:25:46.562250 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.562238    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f85727e-2a60-41dc-8000-9be33b6bfdaa" containerName="revoke-keys"
Oct 30 19:25:46.567154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.567132    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.586770 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.586747    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:46.586934 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.586794    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:25:46.593165 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.593143    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"]
Oct 30 19:25:46.613552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.613525    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.613687 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.613577    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqtp\" (UniqueName: \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.613778 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.613762    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.714935 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.714896    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.715177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.714956    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.715177 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.715004    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqtp\" (UniqueName: \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.715725 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.715653    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.717607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.717583    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.728841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.728815    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqtp\" (UniqueName: \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\") pod \"maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:46.882261 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:46.882182    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:47.199700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:47.199659    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"]
Oct 30 19:25:47.201457 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:25:47.201427    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212b0745_2e23_4a1d_9ff2_9cf963e2a3bc.slice/crio-9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1 WatchSource:0}: Error finding container 9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1: Status 404 returned error can't find the container with id 9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1
Oct 30 19:25:47.931115 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:47.931058    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p" event={"ID":"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc","Type":"ContainerStarted","Data":"51668ebe32d57753864384eb446dc97563a23514d952799793c0d2805771a634"}
Oct 30 19:25:47.931540 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:47.931129    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p" event={"ID":"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc","Type":"ContainerStarted","Data":"9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1"}
Oct 30 19:25:47.957997 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:47.957952    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p" podStartSLOduration=1.957939251 podStartE2EDuration="1.957939251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:47.954427579 +0000 UTC m=+3168.217892542" watchObservedRunningTime="2026-10-30 19:25:47.957939251 +0000 UTC m=+3168.221404199"
Oct 30 19:25:48.940180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:48.940137    5065 generic.go:356] "Generic (PLEG): container finished" podID="212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" containerID="51668ebe32d57753864384eb446dc97563a23514d952799793c0d2805771a634" exitCode=0
Oct 30 19:25:48.941055 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:48.940177    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p" event={"ID":"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc","Type":"ContainerDied","Data":"51668ebe32d57753864384eb446dc97563a23514d952799793c0d2805771a634"}
Oct 30 19:25:50.092333 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.092308    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:50.152268 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.152231    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\") pod \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") "
Oct 30 19:25:50.152462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.152383    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\") pod \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") "
Oct 30 19:25:50.152545 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.152520    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\" (UniqueName: \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\") pod \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\" (UID: \"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc\") "
Oct 30 19:25:50.152660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.152631    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca" pod "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" (UID: "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:50.153203 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.153028    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:50.154763 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.154732    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca" pod "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" (UID: "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:50.155459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.155435    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp" pod "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" (UID: "212b0745-2e23-4a1d-9ff2-9cf963e2a3bc"). InnerVolumeSpecName "kube-api-access-pfqtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:50.263408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.255394    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:50.263408 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.255441    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfqtp\" (UniqueName: \"kubernetes.io/projected/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc-kube-api-access-pfqtp\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:50.957124 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.957065    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"
Oct 30 19:25:50.957300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.957115    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p" event={"ID":"212b0745-2e23-4a1d-9ff2-9cf963e2a3bc","Type":"ContainerDied","Data":"9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1"}
Oct 30 19:25:50.957300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:25:50.957172    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c92a5b4a90ae5f39bcda1b53061620b34d59cb1daff44f2afcb818e0e7b86f1"
Oct 30 19:25:53.056962 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:25:53.056926    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:26:00.458051 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:00.458020    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"]
Oct 30 19:26:00.463226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:00.463203    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-58bb38-mhz8p"]
Oct 30 19:26:01.890468 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:01.890430    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" path="/var/lib/kubelet/pods/212b0745-2e23-4a1d-9ff2-9cf963e2a3bc/volumes"
Oct 30 19:26:10.030955 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.030922    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"]
Oct 30 19:26:10.031415 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.031401    5065 cpu_manager.go:414] "containerMap: removing container" podUID="212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" containerName="revoke-keys"
Oct 30 19:26:10.031467 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.031416    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" containerName="revoke-keys"
Oct 30 19:26:10.031526 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.031518    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="212b0745-2e23-4a1d-9ff2-9cf963e2a3bc" containerName="revoke-keys"
Oct 30 19:26:10.035058 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.035037    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.043395 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.043372    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:10.045932 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.045910    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:26:10.049745 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.049718    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"]
Oct 30 19:26:10.197075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.197042    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmwm\" (UniqueName: \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.197299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.197122    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.197299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.197229    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.298040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.297966    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmwm\" (UniqueName: \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.298040 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.298019    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.298294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.298207    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.298796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.298771    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.300627 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.300602    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.311400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.311358    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmwm\" (UniqueName: \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\") pod \"maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.353360 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.353333    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:10.643931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:10.643903    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"]
Oct 30 19:26:10.645815 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:26:10.645779    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cfa1f9_b578_4769_a24f_0ba4691cdb9a.slice/crio-f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f WatchSource:0}: Error finding container f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f: Status 404 returned error can't find the container with id f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f
Oct 30 19:26:11.120026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:11.119992    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r" event={"ID":"89cfa1f9-b578-4769-a24f-0ba4691cdb9a","Type":"ContainerStarted","Data":"f289dc6fe2777149235516339c4980770152eb326287aa10cfeaeb5bb5ee8253"}
Oct 30 19:26:11.120503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:11.120044    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r" event={"ID":"89cfa1f9-b578-4769-a24f-0ba4691cdb9a","Type":"ContainerStarted","Data":"f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f"}
Oct 30 19:26:11.148941 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:11.148846    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r" podStartSLOduration=1.148831954 podStartE2EDuration="1.148831954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:11.147064602 +0000 UTC m=+3191.410529551" watchObservedRunningTime="2026-10-30 19:26:11.148831954 +0000 UTC m=+3191.412296902"
Oct 30 19:26:12.130009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:12.129976    5065 generic.go:356] "Generic (PLEG): container finished" podID="89cfa1f9-b578-4769-a24f-0ba4691cdb9a" containerID="f289dc6fe2777149235516339c4980770152eb326287aa10cfeaeb5bb5ee8253" exitCode=0
Oct 30 19:26:12.130512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:12.130049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r" event={"ID":"89cfa1f9-b578-4769-a24f-0ba4691cdb9a","Type":"ContainerDied","Data":"f289dc6fe2777149235516339c4980770152eb326287aa10cfeaeb5bb5ee8253"}
Oct 30 19:26:13.292340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.292307    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:13.332118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.329380    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\" (UniqueName: \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\") pod \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") "
Oct 30 19:26:13.332118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.329897    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\") pod \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") "
Oct 30 19:26:13.332118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.329959    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\") pod \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\" (UID: \"89cfa1f9-b578-4769-a24f-0ba4691cdb9a\") "
Oct 30 19:26:13.332118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.330291    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca" pod "89cfa1f9-b578-4769-a24f-0ba4691cdb9a" (UID: "89cfa1f9-b578-4769-a24f-0ba4691cdb9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:13.332118 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.330551    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:13.332691 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.332667    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm" pod "89cfa1f9-b578-4769-a24f-0ba4691cdb9a" (UID: "89cfa1f9-b578-4769-a24f-0ba4691cdb9a"). InnerVolumeSpecName "kube-api-access-tzmwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:13.332827 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.332713    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca" pod "89cfa1f9-b578-4769-a24f-0ba4691cdb9a" (UID: "89cfa1f9-b578-4769-a24f-0ba4691cdb9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:13.431672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.431596    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzmwm\" (UniqueName: \"kubernetes.io/projected/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-kube-api-access-tzmwm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:13.431672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:13.431625    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfa1f9-b578-4769-a24f-0ba4691cdb9a-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.147153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:14.147043    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"
Oct 30 19:26:14.147153 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:14.147065    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r" event={"ID":"89cfa1f9-b578-4769-a24f-0ba4691cdb9a","Type":"ContainerDied","Data":"f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f"}
Oct 30 19:26:14.147400 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:14.147155    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88562db15951f0dc0af52a4663e5d6e582e5e2398dc52747195d25382e3036f"
Oct 30 19:26:16.710793 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.710760    5065 scope.go:122] "RemoveContainer" containerID="a7060eddec4a389961e67737c4c82d6883b062abf7070204ccfd165cb77fffdf"
Oct 30 19:26:16.721372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.721346    5065 scope.go:122] "RemoveContainer" containerID="6b1cd94a9cee2b77da4919e024dd43db02388a3e22635a223250346e0bc4814b"
Oct 30 19:26:16.731788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.731769    5065 scope.go:122] "RemoveContainer" containerID="c4961519a6fe0c03952f8fffdec08d59fc6e7b1dfc5db748d6b541f0f601328f"
Oct 30 19:26:16.740855 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.740839    5065 scope.go:122] "RemoveContainer" containerID="6dc37c8c4ebb35247c63ca5dc06a5a4be3db223d80737f5b5edf267dfd4335bf"
Oct 30 19:26:16.749579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.749562    5065 scope.go:122] "RemoveContainer" containerID="4bcf30ca91bdb9d1ae70ca42a41b3745dce733257218305cd34e998ce65f6587"
Oct 30 19:26:16.758339 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:16.758322    5065 scope.go:122] "RemoveContainer" containerID="9c5246bd3374ca8cb8deff0f6dc39b562ee94243f846ba250f40a9bb5216ee37"
Oct 30 19:26:20.042894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:20.042850    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"]
Oct 30 19:26:20.046779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:20.046751    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e3d1-z7n6n"]
Oct 30 19:26:20.174161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:20.174129    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:26:20.196123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:20.196071    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:26:21.889508 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:21.889473    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5692a780-8fdd-484f-a30c-d127de618e4a" path="/var/lib/kubelet/pods/5692a780-8fdd-484f-a30c-d127de618e4a/volumes"
Oct 30 19:26:23.064016 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:26:23.063981    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:26:24.870704 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:24.870673    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"]
Oct 30 19:26:24.873762 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:24.873738    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a717c7-trj5r"]
Oct 30 19:26:25.174648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:25.174565    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing" containerID="cri-o://cf3a042c405d28b1f9a4bd5f1119b941c3e5606a18d3e9c6a95b47595a4ce04b" gracePeriod=25
Oct 30 19:26:25.197234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:25.197202    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing" containerID="cri-o://38e2a8da6e7a17e21a58f59cb9ce596381762eae9675e25428a75a4f3e3e3913" gracePeriod=25
Oct 30 19:26:25.889461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:25.889432    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cfa1f9-b578-4769-a24f-0ba4691cdb9a" path="/var/lib/kubelet/pods/89cfa1f9-b578-4769-a24f-0ba4691cdb9a/volumes"
Oct 30 19:26:30.026896 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:30.026860    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"]
Oct 30 19:26:30.029962 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:30.029938    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b345b4-qrq69"]
Oct 30 19:26:31.026356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:31.026324    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"]
Oct 30 19:26:31.029303 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:31.029280    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0248f2d1-w745z"]
Oct 30 19:26:31.890655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:31.890612    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a" path="/var/lib/kubelet/pods/aaf8b8c5-fe83-438d-9fb9-7aa65f780b6a/volumes"
Oct 30 19:26:31.891200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:31.891182    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cebd859c-e39c-4350-a294-7f99ddcedc61" path="/var/lib/kubelet/pods/cebd859c-e39c-4350-a294-7f99ddcedc61/volumes"
Oct 30 19:26:32.081424 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:32.081387    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"]
Oct 30 19:26:32.096453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:32.096427    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01bda1-7g5ff"]
Oct 30 19:26:32.933657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:32.933625    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:26:32.934278 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:32.933855    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy" containerID="cri-o://7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657" gracePeriod=30
Oct 30 19:26:33.514615 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.514578    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"]
Oct 30 19:26:33.515148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.515131    5065 cpu_manager.go:414] "containerMap: removing container" podUID="89cfa1f9-b578-4769-a24f-0ba4691cdb9a" containerName="revoke-keys"
Oct 30 19:26:33.515148 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.515147    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cfa1f9-b578-4769-a24f-0ba4691cdb9a" containerName="revoke-keys"
Oct 30 19:26:33.515254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.515248    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cfa1f9-b578-4769-a24f-0ba4691cdb9a" containerName="revoke-keys"
Oct 30 19:26:33.519796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.519777    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.527833 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.527809    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"]
Oct 30 19:26:33.531841 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.531816    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmcz\" (UniqueName: \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.531952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.531866    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.532054 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.532030    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.586937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.586910    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:26:33.587097 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.586939    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:33.632965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.632928    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmcz\" (UniqueName: \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.633157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.632975    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.633157 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.633033    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.633721 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.633690    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.635577 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.635555    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.646963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.646939    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmcz\" (UniqueName: \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\") pod \"maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.842692 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.842609    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:33.892847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:33.892814    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08148bce-5390-4fbb-8327-e40781290654" path="/var/lib/kubelet/pods/08148bce-5390-4fbb-8327-e40781290654/volumes"
Oct 30 19:26:34.158571 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:26:34.158532    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ec54af_b3e4_4a0b_9130_9e872be23ec8.slice/crio-35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef WatchSource:0}: Error finding container 35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef: Status 404 returned error can't find the container with id 35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef
Oct 30 19:26:34.158571 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:34.158543    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"]
Oct 30 19:26:34.223928 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:34.223900    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:34.224046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:34.223901    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:34.309901 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:34.309876    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m" event={"ID":"45ec54af-b3e4-4a0b-9130-9e872be23ec8","Type":"ContainerStarted","Data":"35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef"}
Oct 30 19:26:35.320281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:35.320241    5065 generic.go:356] "Generic (PLEG): container finished" podID="45ec54af-b3e4-4a0b-9130-9e872be23ec8" containerID="0c8fd39c7204391368ad2e569247f428773839612af24e1c00bf2d2adf9a8af3" exitCode=0
Oct 30 19:26:35.320694 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:35.320329    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m" event={"ID":"45ec54af-b3e4-4a0b-9130-9e872be23ec8","Type":"ContainerDied","Data":"0c8fd39c7204391368ad2e569247f428773839612af24e1c00bf2d2adf9a8af3"}
Oct 30 19:26:36.478861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.478839    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:36.567624 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.567581    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\" (UniqueName: \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\") pod \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") "
Oct 30 19:26:36.567846 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.567734    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\") pod \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") "
Oct 30 19:26:36.568174 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.568151    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca" pod "45ec54af-b3e4-4a0b-9130-9e872be23ec8" (UID: "45ec54af-b3e4-4a0b-9130-9e872be23ec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:36.570030 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.569996    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz" pod "45ec54af-b3e4-4a0b-9130-9e872be23ec8" (UID: "45ec54af-b3e4-4a0b-9130-9e872be23ec8"). InnerVolumeSpecName "kube-api-access-pcmcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:36.669200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.669172    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\") pod \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\" (UID: \"45ec54af-b3e4-4a0b-9130-9e872be23ec8\") "
Oct 30 19:26:36.669657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.669632    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:36.669657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.669657    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmcz\" (UniqueName: \"kubernetes.io/projected/45ec54af-b3e4-4a0b-9130-9e872be23ec8-kube-api-access-pcmcz\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:36.671356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.671332    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca" pod "45ec54af-b3e4-4a0b-9130-9e872be23ec8" (UID: "45ec54af-b3e4-4a0b-9130-9e872be23ec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:36.770673 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:36.770648    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ec54af-b3e4-4a0b-9130-9e872be23ec8-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:37.336009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:37.335976    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"
Oct 30 19:26:37.336009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:37.336006    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m" event={"ID":"45ec54af-b3e4-4a0b-9130-9e872be23ec8","Type":"ContainerDied","Data":"35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef"}
Oct 30 19:26:37.336311 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:37.336048    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35a3660e5f7da57a29307babb69c32637ce7987f9bcfc78f7b2c9ecdc921d1ef"
Oct 30 19:26:38.206929 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.206906    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:26:38.284882 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.284850    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.284894    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.284935    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285060 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.284971    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285120    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285206    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:38.285290 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285221    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285311    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285344    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285380    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\") pod \"a047c669-648f-4d9b-86ef-866441a1330b\" (UID: \"a047c669-648f-4d9b-86ef-866441a1330b\") "
Oct 30 19:26:38.285638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285471    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:38.285638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285484    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:38.285788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285765    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:38.285788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285768    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:38.285913 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285891    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.285993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285918    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.285993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285932    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.285993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285946    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a047c669-648f-4d9b-86ef-866441a1330b-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.285993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.285958    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.287891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.287866    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:38.288012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.287899    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "kube-api-access-tktlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:38.288128 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.288073    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:26:38.288207 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.288074    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy" pod "a047c669-648f-4d9b-86ef-866441a1330b" (UID: "a047c669-648f-4d9b-86ef-866441a1330b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:38.344466 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.344437    5065 generic.go:356] "Generic (PLEG): container finished" podID="a047c669-648f-4d9b-86ef-866441a1330b" containerID="7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657" exitCode=0
Oct 30 19:26:38.344576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.344488    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"
Oct 30 19:26:38.344576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.344523    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerDied","Data":"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"}
Oct 30 19:26:38.344689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.344580    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn" event={"ID":"a047c669-648f-4d9b-86ef-866441a1330b","Type":"ContainerDied","Data":"14b44b96579454965a9fd4050e11c7fb4e52dcdbde0ada0b00d4cef206b06ba6"}
Oct 30 19:26:38.344689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.344606    5065 scope.go:122] "RemoveContainer" containerID="7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"
Oct 30 19:26:38.356521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.356506    5065 scope.go:122] "RemoveContainer" containerID="52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"
Oct 30 19:26:38.366718 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.366699    5065 scope.go:122] "RemoveContainer" containerID="7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"
Oct 30 19:26:38.367005 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:26:38.366985    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657\": container with ID starting with 7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657 not found: ID does not exist" containerID="7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"
Oct 30 19:26:38.367075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.367012    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657"} err="failed to get container status \"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657\": rpc error: code = NotFound desc = could not find container \"7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657\": container with ID starting with 7ced70fa3790c6d9a1d4bb59ec06f1aab3a0e090d26f11a853d44c2a7c080657 not found: ID does not exist"
Oct 30 19:26:38.367075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.367028    5065 scope.go:122] "RemoveContainer" containerID="52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"
Oct 30 19:26:38.367288 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:26:38.367274    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac\": container with ID starting with 52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac not found: ID does not exist" containerID="52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"
Oct 30 19:26:38.367338 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.367292    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac"} err="failed to get container status \"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac\": rpc error: code = NotFound desc = could not find container \"52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac\": container with ID starting with 52f60a92e0c1096889226285f5a20bb1ef4a5678bfcac2761ac636b8656f7bac not found: ID does not exist"
Oct 30 19:26:38.372752 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.372723    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:26:38.377457 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.377437    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a717c7-openshift-default-5f44cc795c-rnwvn"]
Oct 30 19:26:38.386923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.386905    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a047c669-648f-4d9b-86ef-866441a1330b-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.387004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.386925    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a047c669-648f-4d9b-86ef-866441a1330b-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.387004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.386935    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tktlg\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-kube-api-access-tktlg\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:38.387004 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:38.386946    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a047c669-648f-4d9b-86ef-866441a1330b-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:39.888527 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:39.888491    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a047c669-648f-4d9b-86ef-866441a1330b" path="/var/lib/kubelet/pods/a047c669-648f-4d9b-86ef-866441a1330b/volumes"
Oct 30 19:26:43.028740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.028700    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"]
Oct 30 19:26:43.033963 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.033936    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a84ebc-x5v89"]
Oct 30 19:26:43.036699 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.036668    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"]
Oct 30 19:26:43.040697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.040673    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d29fe69-qzlfc"]
Oct 30 19:26:43.889216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.889186    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621023c3-d46d-48ca-9e22-dc7ce4505af0" path="/var/lib/kubelet/pods/621023c3-d46d-48ca-9e22-dc7ce4505af0/volumes"
Oct 30 19:26:43.889662 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:43.889646    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b01866-a6da-4939-9bdb-83d970b07ef6" path="/var/lib/kubelet/pods/63b01866-a6da-4939-9bdb-83d970b07ef6/volumes"
Oct 30 19:26:44.224272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:44.224198    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:44.224272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:44.224210    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:44.537074 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:44.537041    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"]
Oct 30 19:26:44.539818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:44.539791    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f7c6e3-4jq4m"]
Oct 30 19:26:45.412714 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.412673    5065 generic.go:356] "Generic (PLEG): container finished" podID="6ecf4571-f593-450f-b159-8c256405fdb0" containerID="38e2a8da6e7a17e21a58f59cb9ce596381762eae9675e25428a75a4f3e3e3913" exitCode=0
Oct 30 19:26:45.413474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.413426    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" event={"ID":"6ecf4571-f593-450f-b159-8c256405fdb0","Type":"ContainerDied","Data":"38e2a8da6e7a17e21a58f59cb9ce596381762eae9675e25428a75a4f3e3e3913"}
Oct 30 19:26:45.416612 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.416579    5065 generic.go:356] "Generic (PLEG): container finished" podID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerID="cf3a042c405d28b1f9a4bd5f1119b941c3e5606a18d3e9c6a95b47595a4ce04b" exitCode=0
Oct 30 19:26:45.416893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.416867    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" event={"ID":"3cf8a994-50f0-4224-afa7-1715b4527ad0","Type":"ContainerDied","Data":"cf3a042c405d28b1f9a4bd5f1119b941c3e5606a18d3e9c6a95b47595a4ce04b"}
Oct 30 19:26:45.498262 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.498236    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:26:45.513441 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.513420    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:26:45.560693 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.560642    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\") pod \"6ecf4571-f593-450f-b159-8c256405fdb0\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") "
Oct 30 19:26:45.560891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.560703    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\") pod \"3cf8a994-50f0-4224-afa7-1715b4527ad0\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") "
Oct 30 19:26:45.560891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.560849    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\" (UniqueName: \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\") pod \"6ecf4571-f593-450f-b159-8c256405fdb0\" (UID: \"6ecf4571-f593-450f-b159-8c256405fdb0\") "
Oct 30 19:26:45.560891 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.560887    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\" (UniqueName: \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\") pod \"3cf8a994-50f0-4224-afa7-1715b4527ad0\" (UID: \"3cf8a994-50f0-4224-afa7-1715b4527ad0\") "
Oct 30 19:26:45.564651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.562719    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume" pod "6ecf4571-f593-450f-b159-8c256405fdb0" (UID: "6ecf4571-f593-450f-b159-8c256405fdb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:45.564651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.562966    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume" pod "3cf8a994-50f0-4224-afa7-1715b4527ad0" (UID: "3cf8a994-50f0-4224-afa7-1715b4527ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:45.565724 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.565697    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk" pod "6ecf4571-f593-450f-b159-8c256405fdb0" (UID: "6ecf4571-f593-450f-b159-8c256405fdb0"). InnerVolumeSpecName "kube-api-access-csnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:45.573019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.572990    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs" pod "3cf8a994-50f0-4224-afa7-1715b4527ad0" (UID: "3cf8a994-50f0-4224-afa7-1715b4527ad0"). InnerVolumeSpecName "kube-api-access-n4fcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:45.662734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.662708    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf4571-f593-450f-b159-8c256405fdb0-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:45.662734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.662730    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf8a994-50f0-4224-afa7-1715b4527ad0-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:45.662734 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.662740    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnlk\" (UniqueName: \"kubernetes.io/projected/6ecf4571-f593-450f-b159-8c256405fdb0-kube-api-access-csnlk\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:45.662993 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.662748    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4fcs\" (UniqueName: \"kubernetes.io/projected/3cf8a994-50f0-4224-afa7-1715b4527ad0-kube-api-access-n4fcs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:45.889147 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:45.889111    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ec54af-b3e4-4a0b-9130-9e872be23ec8" path="/var/lib/kubelet/pods/45ec54af-b3e4-4a0b-9130-9e872be23ec8/volumes"
Oct 30 19:26:46.425807 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.425768    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5" event={"ID":"6ecf4571-f593-450f-b159-8c256405fdb0","Type":"ContainerDied","Data":"b05004e020acfdfedc62ef41c319db25557893e7e290b49515f405c5c381448e"}
Oct 30 19:26:46.426328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.425841    5065 scope.go:122] "RemoveContainer" containerID="38e2a8da6e7a17e21a58f59cb9ce596381762eae9675e25428a75a4f3e3e3913"
Oct 30 19:26:46.426328 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.425777    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"
Oct 30 19:26:46.427683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.427634    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"
Oct 30 19:26:46.427862 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.427833    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj" event={"ID":"3cf8a994-50f0-4224-afa7-1715b4527ad0","Type":"ContainerDied","Data":"21d26de98efc0c06109b49cf849845eb4f66751723c089fcef22279ccaa32404"}
Oct 30 19:26:46.438196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.438173    5065 scope.go:122] "RemoveContainer" containerID="cf3a042c405d28b1f9a4bd5f1119b941c3e5606a18d3e9c6a95b47595a4ce04b"
Oct 30 19:26:46.469634 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.469611    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:26:46.477202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.477176    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a717c7-cf6d9955f-cn8p5"]
Oct 30 19:26:46.506698 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.506662    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:26:46.511122 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:46.511072    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a717c7-87787c488-ndsdj"]
Oct 30 19:26:47.889774 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:47.889733    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" path="/var/lib/kubelet/pods/3cf8a994-50f0-4224-afa7-1715b4527ad0/volumes"
Oct 30 19:26:47.890307 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:47.890178    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" path="/var/lib/kubelet/pods/6ecf4571-f593-450f-b159-8c256405fdb0/volumes"
Oct 30 19:26:53.060034 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:26:53.059990    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:26:53.168178 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168145    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"]
Oct 30 19:26:53.168651 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168638    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:26:53.168700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168656    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:26:53.168700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168674    5065 cpu_manager.go:414] "containerMap: removing container" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing"
Oct 30 19:26:53.168700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168679    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing"
Oct 30 19:26:53.168700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168698    5065 cpu_manager.go:414] "containerMap: removing container" podUID="45ec54af-b3e4-4a0b-9130-9e872be23ec8" containerName="revoke-keys"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168703    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ec54af-b3e4-4a0b-9130-9e872be23ec8" containerName="revoke-keys"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168710    5065 cpu_manager.go:414] "containerMap: removing container" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168716    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168725    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168733    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing"
Oct 30 19:26:53.168836 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168830    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:26:53.169032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168840    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf8a994-50f0-4224-afa7-1715b4527ad0" containerName="payload-processing"
Oct 30 19:26:53.169032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168849    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ec54af-b3e4-4a0b-9130-9e872be23ec8" containerName="revoke-keys"
Oct 30 19:26:53.169032 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.168855    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf4571-f593-450f-b159-8c256405fdb0" containerName="payload-pre-processing"
Oct 30 19:26:53.174141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.174118    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.182335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.182306    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:53.182582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.182563    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:26:53.185159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.184454    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"]
Oct 30 19:26:53.238364 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.238332    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.238524 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.238403    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmwn\" (UniqueName: \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.238578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.238521    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.339249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.339169    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmwn\" (UniqueName: \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.339249 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.339229    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.339463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.339339    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.340012 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.339988    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.341982 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.341957    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.352923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.352901    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmwn\" (UniqueName: \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.492592 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.492562    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:53.804236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:53.804208    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"]
Oct 30 19:26:53.807334 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:26:53.807301    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a248ea_3107_4803_9156_702ebbf42370.slice/crio-5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc WatchSource:0}: Error finding container 5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc: Status 404 returned error can't find the container with id 5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc
Oct 30 19:26:54.494076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:54.493976    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c" event={"ID":"10a248ea-3107-4803-9156-702ebbf42370","Type":"ContainerStarted","Data":"595a7fe43005b6bbf49442fd33b64e53f311107613bbb11b8aee698fd5f3a884"}
Oct 30 19:26:54.494076 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:54.494049    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c" event={"ID":"10a248ea-3107-4803-9156-702ebbf42370","Type":"ContainerStarted","Data":"5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc"}
Oct 30 19:26:54.519994 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:54.519941    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c" podStartSLOduration=1.5199266150000001 podStartE2EDuration="1.519926615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:54.515722335 +0000 UTC m=+3234.779187287" watchObservedRunningTime="2026-10-30 19:26:54.519926615 +0000 UTC m=+3234.783391569"
Oct 30 19:26:55.505864 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:55.505826    5065 generic.go:356] "Generic (PLEG): container finished" podID="10a248ea-3107-4803-9156-702ebbf42370" containerID="595a7fe43005b6bbf49442fd33b64e53f311107613bbb11b8aee698fd5f3a884" exitCode=0
Oct 30 19:26:55.506357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:55.505887    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c" event={"ID":"10a248ea-3107-4803-9156-702ebbf42370","Type":"ContainerDied","Data":"595a7fe43005b6bbf49442fd33b64e53f311107613bbb11b8aee698fd5f3a884"}
Oct 30 19:26:56.664336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.664311    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:56.782586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.782503    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\") pod \"10a248ea-3107-4803-9156-702ebbf42370\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") "
Oct 30 19:26:56.782586 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.782574    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\") pod \"10a248ea-3107-4803-9156-702ebbf42370\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") "
Oct 30 19:26:56.782832 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.782635    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\" (UniqueName: \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\") pod \"10a248ea-3107-4803-9156-702ebbf42370\" (UID: \"10a248ea-3107-4803-9156-702ebbf42370\") "
Oct 30 19:26:56.782960 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.782936    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca" pod "10a248ea-3107-4803-9156-702ebbf42370" (UID: "10a248ea-3107-4803-9156-702ebbf42370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:56.783057 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.783036    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a248ea-3107-4803-9156-702ebbf42370-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:56.784968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.784945    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca" pod "10a248ea-3107-4803-9156-702ebbf42370" (UID: "10a248ea-3107-4803-9156-702ebbf42370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:56.785162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.785075    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn" pod "10a248ea-3107-4803-9156-702ebbf42370" (UID: "10a248ea-3107-4803-9156-702ebbf42370"). InnerVolumeSpecName "kube-api-access-vsmwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:56.883723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.883690    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmwn\" (UniqueName: \"kubernetes.io/projected/10a248ea-3107-4803-9156-702ebbf42370-kube-api-access-vsmwn\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:56.883723 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:56.883718    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a248ea-3107-4803-9156-702ebbf42370-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:57.521738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:57.521706    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"
Oct 30 19:26:57.521738 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:57.521726    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c" event={"ID":"10a248ea-3107-4803-9156-702ebbf42370","Type":"ContainerDied","Data":"5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc"}
Oct 30 19:26:57.521977 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:26:57.521779    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc68e53e4c24125219f67eaa6b8ea0d2f8880b9617ec506a4d6b461ee1542dc"
Oct 30 19:27:06.376230 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:06.376191    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"]
Oct 30 19:27:06.379017 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:06.378989    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e1e1ff-wsr5c"]
Oct 30 19:27:07.913969 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:07.913917    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a248ea-3107-4803-9156-702ebbf42370" path="/var/lib/kubelet/pods/10a248ea-3107-4803-9156-702ebbf42370/volumes"
Oct 30 19:27:10.028503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:10.028450    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"]
Oct 30 19:27:10.031897 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:10.031868    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e9bdd6-vn8jf"]
Oct 30 19:27:11.891187 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:11.891154    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c" path="/var/lib/kubelet/pods/2b37d90f-b5af-4ebd-a774-3f57e0bdaa4c/volumes"
Oct 30 19:27:13.338282 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.338248    5065 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Oct 30 19:27:13.338740 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.338332    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Oct 30 19:27:13.672504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.672418    5065 generic.go:356] "Generic (PLEG): container finished" podID="64c55551-f528-4226-b624-6a9f8409b0f6" containerID="a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4" exitCode=0
Oct 30 19:27:13.672504 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.672483    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerDied","Data":"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"}
Oct 30 19:27:13.672834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.672816    5065 scope.go:122] "RemoveContainer" containerID="a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"
Oct 30 19:27:13.674127 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.674060    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:13.674244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.674183    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:13.674244 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:13.674233    5065 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:14.029502 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.029465    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"]
Oct 30 19:27:14.032279 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.032256    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"]
Oct 30 19:27:14.034334 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.034308    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c1d4c1-c58nt"]
Oct 30 19:27:14.036594 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.036569    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114e9063-dv42l"]
Oct 30 19:27:14.402159 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.397279    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:27:14.683197 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.683115    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerStarted","Data":"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"}
Oct 30 19:27:14.683366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:14.683278    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy" containerID="cri-o://1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283" gracePeriod=30
Oct 30 19:27:15.013393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.013364    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"]
Oct 30 19:27:15.013922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.013903    5065 cpu_manager.go:414] "containerMap: removing container" podUID="10a248ea-3107-4803-9156-702ebbf42370" containerName="revoke-keys"
Oct 30 19:27:15.014029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.013924    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a248ea-3107-4803-9156-702ebbf42370" containerName="revoke-keys"
Oct 30 19:27:15.014140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.014036    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a248ea-3107-4803-9156-702ebbf42370" containerName="revoke-keys"
Oct 30 19:27:15.014140 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.014054    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="a047c669-648f-4d9b-86ef-866441a1330b" containerName="istio-proxy"
Oct 30 19:27:15.017391 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.017372    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.024242 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.024218    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:15.024342 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.024287    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:27:15.037305 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.037285    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"]
Oct 30 19:27:15.202265 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.202228    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.202422 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.202374    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.202511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.202459    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fr97\" (UniqueName: \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.303648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.303572    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.303648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.303613    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fr97\" (UniqueName: \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.303927 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.303670    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.304434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.304412    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.306372 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.306350    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.319113 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.319061    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fr97\" (UniqueName: \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.331868 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.331843    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:15.615226 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.615197    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"]
Oct 30 19:27:15.616839 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:27:15.616807    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d18043_487d_4ae1_93c7_b3f30e55243c.slice/crio-727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf WatchSource:0}: Error finding container 727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf: Status 404 returned error can't find the container with id 727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf
Oct 30 19:27:15.692920 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.692883    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn" event={"ID":"07d18043-487d-4ae1-93c7-b3f30e55243c","Type":"ContainerStarted","Data":"727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf"}
Oct 30 19:27:15.891280 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.891248    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b5c8ab-6c70-417d-b865-7384417f7041" path="/var/lib/kubelet/pods/08b5c8ab-6c70-417d-b865-7384417f7041/volumes"
Oct 30 19:27:15.891750 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:15.891729    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d5e362-913a-49d5-a4cb-981c7a289443" path="/var/lib/kubelet/pods/c6d5e362-913a-49d5-a4cb-981c7a289443/volumes"
Oct 30 19:27:16.702727 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.702643    5065 generic.go:356] "Generic (PLEG): container finished" podID="07d18043-487d-4ae1-93c7-b3f30e55243c" containerID="651f7d8da497b07926dd42d3242de0c4c56002403394b58bf9d8b311b99eb6d0" exitCode=0
Oct 30 19:27:16.703161 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.702722    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn" event={"ID":"07d18043-487d-4ae1-93c7-b3f30e55243c","Type":"ContainerDied","Data":"651f7d8da497b07926dd42d3242de0c4c56002403394b58bf9d8b311b99eb6d0"}
Oct 30 19:27:16.827359 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.827332    5065 scope.go:122] "RemoveContainer" containerID="106ca2078bda4e8d236825052e95b476fa2d48b96824b13b9edc24b9159f2404"
Oct 30 19:27:16.840346 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.840314    5065 scope.go:122] "RemoveContainer" containerID="c1af2e43bed7eb21040342b47e4be1884dd73ff04f5f9719daa39023a241b1cf"
Oct 30 19:27:16.860309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.860272    5065 scope.go:122] "RemoveContainer" containerID="91b0d7496a08ffc51112b92bcf64b3c160094589e3461fefa8581336a2c4c88d"
Oct 30 19:27:16.870697 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.870673    5065 scope.go:122] "RemoveContainer" containerID="b3dc6081f4e01e683f892d8e40778df926c65a3c58643969681a123a3369f8dc"
Oct 30 19:27:16.879923 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.879902    5065 scope.go:122] "RemoveContainer" containerID="5b8c96a83f20d0c09835a87c3e8733652d7b006d5191bdb92e3e44c312ce1df9"
Oct 30 19:27:16.889728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.889511    5065 scope.go:122] "RemoveContainer" containerID="41be79748a77a7cf31e68746764934f7cd5c4841c93080baa19d292479b2673e"
Oct 30 19:27:16.900660 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:16.900639    5065 scope.go:122] "RemoveContainer" containerID="4aebb7189782b4a2513d31755660ac4fb11072d596bace95ec03a7d1b8762d57"
Oct 30 19:27:17.895838 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:17.895809    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:18.033747 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.033710    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\") pod \"07d18043-487d-4ae1-93c7-b3f30e55243c\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") "
Oct 30 19:27:18.033981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.033763    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\") pod \"07d18043-487d-4ae1-93c7-b3f30e55243c\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") "
Oct 30 19:27:18.033981 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.033794    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\" (UniqueName: \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\") pod \"07d18043-487d-4ae1-93c7-b3f30e55243c\" (UID: \"07d18043-487d-4ae1-93c7-b3f30e55243c\") "
Oct 30 19:27:18.034546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.034512    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca" pod "07d18043-487d-4ae1-93c7-b3f30e55243c" (UID: "07d18043-487d-4ae1-93c7-b3f30e55243c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:18.036800 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.036756    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca" pod "07d18043-487d-4ae1-93c7-b3f30e55243c" (UID: "07d18043-487d-4ae1-93c7-b3f30e55243c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:18.037142 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.036866    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97" pod "07d18043-487d-4ae1-93c7-b3f30e55243c" (UID: "07d18043-487d-4ae1-93c7-b3f30e55243c"). InnerVolumeSpecName "kube-api-access-5fr97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:18.135202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.135170    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:18.135202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.135198    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d18043-487d-4ae1-93c7-b3f30e55243c-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:18.135202 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.135208    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fr97\" (UniqueName: \"kubernetes.io/projected/07d18043-487d-4ae1-93c7-b3f30e55243c-kube-api-access-5fr97\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:18.732300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.732264    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"
Oct 30 19:27:18.732509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.732269    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn" event={"ID":"07d18043-487d-4ae1-93c7-b3f30e55243c","Type":"ContainerDied","Data":"727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf"}
Oct 30 19:27:18.732509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:18.732437    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727318a2f8e120ada05ad9ce988dd7e6fbe627399d9b0b1edc8d6717f8ddb7cf"
Oct 30 19:27:19.966902 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:19.966875    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:27:20.052858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.052833    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.052887    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.052942    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.052971    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053024 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053003    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053053    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053080    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.053393 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053339    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.053569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053459    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.053569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053451    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:20.053569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.053532    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.055322 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.055296    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:20.055322 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.055296    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:27:20.055701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.055664    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.055804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.055708    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "kube-api-access-8h85b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:20.154097 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154069    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.154222 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154143    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\") pod \"64c55551-f528-4226-b624-6a9f8409b0f6\" (UID: \"64c55551-f528-4226-b624-6a9f8409b0f6\") "
Oct 30 19:27:20.154287 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154249    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.154384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154365    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket" pod "64c55551-f528-4226-b624-6a9f8409b0f6" (UID: "64c55551-f528-4226-b624-6a9f8409b0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.154542 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154529    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154546    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154562    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h85b\" (UniqueName: \"kubernetes.io/projected/64c55551-f528-4226-b624-6a9f8409b0f6-kube-api-access-8h85b\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154571    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c55551-f528-4226-b624-6a9f8409b0f6-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154584 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154579    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c55551-f528-4226-b624-6a9f8409b0f6-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154586    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154594    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.154715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.154602    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c55551-f528-4226-b624-6a9f8409b0f6-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:20.746840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.746806    5065 generic.go:356] "Generic (PLEG): container finished" podID="64c55551-f528-4226-b624-6a9f8409b0f6" containerID="1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283" exitCode=0
Oct 30 19:27:20.747018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.746853    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"
Oct 30 19:27:20.747018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.746888    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerDied","Data":"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"}
Oct 30 19:27:20.747018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.746948    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k" event={"ID":"64c55551-f528-4226-b624-6a9f8409b0f6","Type":"ContainerDied","Data":"633f806644c8131b7c26d018e494953bb425db7e14ee279b5b3ede76f1a47cf9"}
Oct 30 19:27:20.747018 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.746979    5065 scope.go:122] "RemoveContainer" containerID="1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"
Oct 30 19:27:20.758354 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.758333    5065 scope.go:122] "RemoveContainer" containerID="a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"
Oct 30 19:27:20.768878 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.768856    5065 scope.go:122] "RemoveContainer" containerID="1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"
Oct 30 19:27:20.769282 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:20.769258    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283\": container with ID starting with 1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283 not found: ID does not exist" containerID="1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"
Oct 30 19:27:20.769398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.769292    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283"} err="failed to get container status \"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283\": rpc error: code = NotFound desc = could not find container \"1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283\": container with ID starting with 1d30b6a740991095d7b39fa3e7924788a8f71f39c75df317f4c00cf92f105283 not found: ID does not exist"
Oct 30 19:27:20.769398 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.769317    5065 scope.go:122] "RemoveContainer" containerID="a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"
Oct 30 19:27:20.769645 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:20.769621    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4\": container with ID starting with a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4 not found: ID does not exist" containerID="a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"
Oct 30 19:27:20.769706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.769657    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4"} err="failed to get container status \"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4\": rpc error: code = NotFound desc = could not find container \"a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4\": container with ID starting with a8068afb8355e8f198a212f342d9794f974eacf6b113d233d37f3b6228fca0e4 not found: ID does not exist"
Oct 30 19:27:20.773248 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.773223    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:27:20.778742 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:20.778722    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e1e1ff-openshift-default-68c4cd94f5-j5b9k"]
Oct 30 19:27:21.026945 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:21.026922    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"]
Oct 30 19:27:21.030968 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:21.030946    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa00328-l4k7s"]
Oct 30 19:27:21.887566 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:21.887534    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d4e518-56e0-4938-bc8e-63a380297efa" path="/var/lib/kubelet/pods/27d4e518-56e0-4938-bc8e-63a380297efa/volumes"
Oct 30 19:27:21.887973 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:21.887957    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" path="/var/lib/kubelet/pods/64c55551-f528-4226-b624-6a9f8409b0f6/volumes"
Oct 30 19:27:23.056204 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:23.056168    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:27:27.228396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:27.228336    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"]
Oct 30 19:27:27.231168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:27.231143    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b0e009-jx4kn"]
Oct 30 19:27:27.889463 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:27.889426    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d18043-487d-4ae1-93c7-b3f30e55243c" path="/var/lib/kubelet/pods/07d18043-487d-4ae1-93c7-b3f30e55243c/volumes"
Oct 30 19:27:35.209190 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:35.209155    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:27:35.209579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:35.209413    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy" containerID="cri-o://8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049" gracePeriod=30
Oct 30 19:27:36.050706 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.050672    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"]
Oct 30 19:27:36.051340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051319    5065 cpu_manager.go:414] "containerMap: removing container" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.051340 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051340    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051350    5065 cpu_manager.go:414] "containerMap: removing container" podUID="07d18043-487d-4ae1-93c7-b3f30e55243c" containerName="revoke-keys"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051357    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d18043-487d-4ae1-93c7-b3f30e55243c" containerName="revoke-keys"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051392    5065 cpu_manager.go:414] "containerMap: removing container" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051397    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051494    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d18043-487d-4ae1-93c7-b3f30e55243c" containerName="revoke-keys"
Oct 30 19:27:36.051506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051503    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.051851 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.051512    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c55551-f528-4226-b624-6a9f8409b0f6" containerName="istio-proxy"
Oct 30 19:27:36.054840 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.054818    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.077850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.077819    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"]
Oct 30 19:27:36.118254 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.118217    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfgw\" (UniqueName: \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.118448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.118353    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.118448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.118435    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.139952 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.139921    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:36.140129 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.140105    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:27:36.219814 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.219788    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.220291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.219863    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.220291 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.219951    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfgw\" (UniqueName: \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.220732 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.220709    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.222491 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.222467    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.233931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.233912    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfgw\" (UniqueName: \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.369258 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.369189    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:36.655025 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.655000    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"]
Oct 30 19:27:36.656527 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:27:36.656500    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6654da81_79ac_46e1_97f2_178d533ebabb.slice/crio-bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b WatchSource:0}: Error finding container bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b: Status 404 returned error can't find the container with id bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b
Oct 30 19:27:36.866037 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:36.866006    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4" event={"ID":"6654da81-79ac-46e1-97f2-178d533ebabb","Type":"ContainerStarted","Data":"bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b"}
Oct 30 19:27:37.874597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:37.874518    5065 generic.go:356] "Generic (PLEG): container finished" podID="6654da81-79ac-46e1-97f2-178d533ebabb" containerID="2aab7528a267ae0996a84e56a682ce3e4cc9b01a228a025043f7688a2abacaf9" exitCode=0
Oct 30 19:27:37.874965 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:37.874610    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4" event={"ID":"6654da81-79ac-46e1-97f2-178d533ebabb","Type":"ContainerDied","Data":"2aab7528a267ae0996a84e56a682ce3e4cc9b01a228a025043f7688a2abacaf9"}
Oct 30 19:27:39.037557 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.037533    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:39.148007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.147943    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\") pod \"6654da81-79ac-46e1-97f2-178d533ebabb\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") "
Oct 30 19:27:39.148007 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.148003    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\") pod \"6654da81-79ac-46e1-97f2-178d533ebabb\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") "
Oct 30 19:27:39.148236 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.148143    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\" (UniqueName: \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\") pod \"6654da81-79ac-46e1-97f2-178d533ebabb\" (UID: \"6654da81-79ac-46e1-97f2-178d533ebabb\") "
Oct 30 19:27:39.148423 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.148398    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca" pod "6654da81-79ac-46e1-97f2-178d533ebabb" (UID: "6654da81-79ac-46e1-97f2-178d533ebabb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:39.150396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.150371    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca" pod "6654da81-79ac-46e1-97f2-178d533ebabb" (UID: "6654da81-79ac-46e1-97f2-178d533ebabb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:39.150396 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.150374    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw" pod "6654da81-79ac-46e1-97f2-178d533ebabb" (UID: "6654da81-79ac-46e1-97f2-178d533ebabb"). InnerVolumeSpecName "kube-api-access-jbfgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:39.248804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.248776    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbfgw\" (UniqueName: \"kubernetes.io/projected/6654da81-79ac-46e1-97f2-178d533ebabb-kube-api-access-jbfgw\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.248804 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.248799    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.248959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.248809    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654da81-79ac-46e1-97f2-178d533ebabb-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:39.891356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.891322    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"
Oct 30 19:27:39.891356 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.891336    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4" event={"ID":"6654da81-79ac-46e1-97f2-178d533ebabb","Type":"ContainerDied","Data":"bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b"}
Oct 30 19:27:39.891648 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:39.891371    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb98b77b53e198a9ea81024b5512826f91238206aed9e4ae362475a0bd5803b"
Oct 30 19:27:40.476162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.476100    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:27:40.565412 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565371    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565453    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565567 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565551    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565654 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565623    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565703 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565667    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565705    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565760 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565735    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565785    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.565829 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.565818    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\") pod \"b0427868-837f-4930-8003-abfe2f9ea3c5\" (UID: \"b0427868-837f-4930-8003-abfe2f9ea3c5\") "
Oct 30 19:27:40.566931 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.566897    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:40.567068 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.566943    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:40.567163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.567078    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:40.567163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.567109    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:40.567288 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.567254    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:40.567822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.567802    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "kube-api-access-v6gxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:40.568983 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.568958    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:27:40.569366 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.569335    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:40.569534 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.569515    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token" pod "b0427868-837f-4930-8003-abfe2f9ea3c5" (UID: "b0427868-837f-4930-8003-abfe2f9ea3c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:40.661180 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.661048    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 19:27:40.666786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666763    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666786 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666787    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666804    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666814    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666823    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gxh\" (UniqueName: \"kubernetes.io/projected/b0427868-837f-4930-8003-abfe2f9ea3c5-kube-api-access-v6gxh\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666833    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0427868-837f-4930-8003-abfe2f9ea3c5-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666842    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666850    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.666933 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.666858    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0427868-837f-4930-8003-abfe2f9ea3c5-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.899876 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.899846    5065 generic.go:356] "Generic (PLEG): container finished" podID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerID="8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049" exitCode=0
Oct 30 19:27:40.900046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.899896    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"
Oct 30 19:27:40.900046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.899928    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerDied","Data":"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"}
Oct 30 19:27:40.900046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.899987    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq" event={"ID":"b0427868-837f-4930-8003-abfe2f9ea3c5","Type":"ContainerDied","Data":"5326f1756386935af1202efeaae76a5dbea3d3dd3a9730749ef499269de9f673"}
Oct 30 19:27:40.900046 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.900016    5065 scope.go:122] "RemoveContainer" containerID="8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"
Oct 30 19:27:40.912191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.912169    5065 scope.go:122] "RemoveContainer" containerID="be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"
Oct 30 19:27:40.922528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.922510    5065 scope.go:122] "RemoveContainer" containerID="8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"
Oct 30 19:27:40.922805 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:40.922778    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049\": container with ID starting with 8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049 not found: ID does not exist" containerID="8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"
Oct 30 19:27:40.922863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.922817    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049"} err="failed to get container status \"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049\": rpc error: code = NotFound desc = could not find container \"8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049\": container with ID starting with 8f4811eb1bdd7620c0890ec7a2687f3b861e172ab33db313556eea7a08c77049 not found: ID does not exist"
Oct 30 19:27:40.922863 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.922839    5065 scope.go:122] "RemoveContainer" containerID="be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"
Oct 30 19:27:40.923200 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:40.923167    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e\": container with ID starting with be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e not found: ID does not exist" containerID="be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"
Oct 30 19:27:40.923299 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.923212    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e"} err="failed to get container status \"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e\": rpc error: code = NotFound desc = could not find container \"be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e\": container with ID starting with be600499feed098307179e8746fb4fba19b7eb959dd20ccf14119b56481f414e not found: ID does not exist"
Oct 30 19:27:40.927039 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.927017    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:27:40.934270 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:40.934251    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b0e009-openshift-default-6dd5c597b9-v5kfq"]
Oct 30 19:27:41.888462 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:41.888433    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" path="/var/lib/kubelet/pods/b0427868-837f-4930-8003-abfe2f9ea3c5/volumes"
Oct 30 19:27:46.141607 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.141575    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 19:27:46.928625 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.928592    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"]
Oct 30 19:27:46.929221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929196    5065 cpu_manager.go:414] "containerMap: removing container" podUID="6654da81-79ac-46e1-97f2-178d533ebabb" containerName="revoke-keys"
Oct 30 19:27:46.929221 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929221    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654da81-79ac-46e1-97f2-178d533ebabb" containerName="revoke-keys"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929240    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929245    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929274    5065 cpu_manager.go:414] "containerMap: removing container" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929284    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929375    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654da81-79ac-46e1-97f2-178d533ebabb" containerName="revoke-keys"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929387    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.929425 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.929396    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0427868-837f-4930-8003-abfe2f9ea3c5" containerName="istio-proxy"
Oct 30 19:27:46.934193 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.934169    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:46.943453 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.943431    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:46.943737 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.943715    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:27:46.949880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:46.949855    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"]
Oct 30 19:27:47.036409 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.036382    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhwf\" (UniqueName: \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.036547 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.036430    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.036668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.036645    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.138151 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.138115    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.138281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.138174    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhwf\" (UniqueName: \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.138281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.138207    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.138849 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.138822    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.140879 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.140860    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.153413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.153392    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhwf\" (UniqueName: \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\") pod \"maas-api-revoke-sub-443a1993-rwj5h\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.257195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.257141    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:47.539847 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.539813    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"]
Oct 30 19:27:47.541145 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:27:47.541121    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ddf00c_585b_4815_af86_67ddccb20b85.slice/crio-778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050 WatchSource:0}: Error finding container 778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050: Status 404 returned error can't find the container with id 778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050
Oct 30 19:27:47.957720 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.957693    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h" event={"ID":"d6ddf00c-585b-4815-af86-67ddccb20b85","Type":"ContainerStarted","Data":"0ff82e63d1e57eb68687380c15e7e5fd3c187afd7770e10cfa756eaff35ef5e9"}
Oct 30 19:27:47.957869 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.957743    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h" event={"ID":"d6ddf00c-585b-4815-af86-67ddccb20b85","Type":"ContainerStarted","Data":"778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050"}
Oct 30 19:27:47.989002 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:47.988949    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h" podStartSLOduration=1.988937591 podStartE2EDuration="1.988937591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:47.986313578 +0000 UTC m=+3288.249778528" watchObservedRunningTime="2026-10-30 19:27:47.988937591 +0000 UTC m=+3288.252402538"
Oct 30 19:27:48.968556 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:48.968512    5065 generic.go:356] "Generic (PLEG): container finished" podID="d6ddf00c-585b-4815-af86-67ddccb20b85" containerID="0ff82e63d1e57eb68687380c15e7e5fd3c187afd7770e10cfa756eaff35ef5e9" exitCode=0
Oct 30 19:27:48.969006 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:48.968564    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h" event={"ID":"d6ddf00c-585b-4815-af86-67ddccb20b85","Type":"ContainerDied","Data":"0ff82e63d1e57eb68687380c15e7e5fd3c187afd7770e10cfa756eaff35ef5e9"}
Oct 30 19:27:50.143070 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.143045    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:50.272162 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.272111    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\" (UniqueName: \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\") pod \"d6ddf00c-585b-4815-af86-67ddccb20b85\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") "
Oct 30 19:27:50.272337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.272193    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\") pod \"d6ddf00c-585b-4815-af86-67ddccb20b85\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") "
Oct 30 19:27:50.272337 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.272296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\") pod \"d6ddf00c-585b-4815-af86-67ddccb20b85\" (UID: \"d6ddf00c-585b-4815-af86-67ddccb20b85\") "
Oct 30 19:27:50.272560 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.272537    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca" pod "d6ddf00c-585b-4815-af86-67ddccb20b85" (UID: "d6ddf00c-585b-4815-af86-67ddccb20b85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:50.272672 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.272658    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-openshift-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:50.274471 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.274435    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca" pod "d6ddf00c-585b-4815-af86-67ddccb20b85" (UID: "d6ddf00c-585b-4815-af86-67ddccb20b85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:50.274617 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.274592    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf" pod "d6ddf00c-585b-4815-af86-67ddccb20b85" (UID: "d6ddf00c-585b-4815-af86-67ddccb20b85"). InnerVolumeSpecName "kube-api-access-khhwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:50.373875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.373840    5065 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ddf00c-585b-4815-af86-67ddccb20b85-maas-api-xks-ca\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:50.373875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.373873    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhwf\" (UniqueName: \"kubernetes.io/projected/d6ddf00c-585b-4815-af86-67ddccb20b85-kube-api-access-khhwf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:50.985472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.985442    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h" event={"ID":"d6ddf00c-585b-4815-af86-67ddccb20b85","Type":"ContainerDied","Data":"778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050"}
Oct 30 19:27:50.985472 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.985467    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-443a1993-rwj5h"
Oct 30 19:27:50.985700 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:50.985491    5065 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778f53cf3b425f68938d4ffed4798a393917e46b4d9c5542e21a38e3bd947050"
Oct 30 19:27:51.142761 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:51.142719    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing" containerID="cri-o://7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef" gracePeriod=25
Oct 30 19:27:53.056285 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:53.056245    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:27:53.967936 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:53.967902    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:55.682378 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:55.682327    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main" containerID="cri-o://06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621" gracePeriod=45
Oct 30 19:27:55.949815 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:55.949789    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:27:56.024191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.024152    5065 generic.go:356] "Generic (PLEG): container finished" podID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerID="06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621" exitCode=0
Oct 30 19:27:56.024191 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.024195    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" event={"ID":"37c74c55-4e95-4d65-bd8d-b36572307f6c","Type":"ContainerDied","Data":"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"}
Oct 30 19:27:56.024433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.024223    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb" event={"ID":"37c74c55-4e95-4d65-bd8d-b36572307f6c","Type":"ContainerDied","Data":"d9528f4bea4189353015574b95c7b6be8d5421643e186e3736a9dad30572b954"}
Oct 30 19:27:56.024433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.024225    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"
Oct 30 19:27:56.024433 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.024238    5065 scope.go:122] "RemoveContainer" containerID="06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"
Oct 30 19:27:56.035506 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.035487    5065 scope.go:122] "RemoveContainer" containerID="06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"
Oct 30 19:27:56.035915 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:27:56.035896    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621\": container with ID starting with 06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621 not found: ID does not exist" containerID="06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"
Oct 30 19:27:56.035991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.035921    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621"} err="failed to get container status \"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621\": rpc error: code = NotFound desc = could not find container \"06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621\": container with ID starting with 06f83d272d33c3daadbff1312a6cbf0819136b620d99adb2058773907aac8621 not found: ID does not exist"
Oct 30 19:27:56.124155 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124128    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124190    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\" (UniqueName: \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124219    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124263    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124336 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124316    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124344    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\") pod \"37c74c55-4e95-4d65-bd8d-b36572307f6c\" (UID: \"37c74c55-4e95-4d65-bd8d-b36572307f6c\") "
Oct 30 19:27:56.124578 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124455    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.124668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124551    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.124668 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124634    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.124798 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124780    5065 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-tmp-dir\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.124861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124800    5065 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-model-cache\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.124861 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.124811    5065 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-home\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.126756 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.126733    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "kube-api-access-29hrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:56.126854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.126748    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:56.126854 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.126786    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm" pod "37c74c55-4e95-4d65-bd8d-b36572307f6c" (UID: "37c74c55-4e95-4d65-bd8d-b36572307f6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.226168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.226117    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hrc\" (UniqueName: \"kubernetes.io/projected/37c74c55-4e95-4d65-bd8d-b36572307f6c-kube-api-access-29hrc\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.226168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.226138    5065 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c74c55-4e95-4d65-bd8d-b36572307f6c-tls-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.226168 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.226149    5065 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c74c55-4e95-4d65-bd8d-b36572307f6c-dshm\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.351387 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.351360    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 19:27:56.354834 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:56.354806    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-w2-8a92cc-kserve-5b46dc8b5-rd6xb"]
Oct 30 19:27:57.888041 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:27:57.888011    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" path="/var/lib/kubelet/pods/37c74c55-4e95-4d65-bd8d-b36572307f6c/volumes"
Oct 30 19:28:02.248569 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:02.248532    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"]
Oct 30 19:28:02.253461 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:02.253430    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8a92cc-s2rt4"]
Oct 30 19:28:03.890294 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:03.890255    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6654da81-79ac-46e1-97f2-178d533ebabb" path="/var/lib/kubelet/pods/6654da81-79ac-46e1-97f2-178d533ebabb/volumes"
Oct 30 19:28:03.968038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:03.968004    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:28:04.056075 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:04.056047    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:28:04.063512 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:04.063490    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:28:04.113575 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:04.113550    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:28:04.120808 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:04.120790    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lr9k_eb49e957-8fde-4701-bbb0-b718a6acd947/ovn-acl-logging/0.log"
Oct 30 19:28:09.446139 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:09.446103    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 19:28:09.446712 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:09.446334    5065 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" containerID="cri-o://e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538" gracePeriod=30
Oct 30 19:28:10.582275 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:10.582240    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf474c468-bdpsm_34502ce6-c71e-47ad-b608-7b50a8214301/authorino/0.log"
Oct 30 19:28:10.665459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:10.665427    5065 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:28:10.665623 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:10.665492    5065 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:28:11.006474 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.006397    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dpdm4_069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4/manager/0.log"
Oct 30 19:28:11.421301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.421275    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 19:28:11.487434 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.487403    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\" (UniqueName: \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\") pod \"35a6702b-966a-4813-875e-c148ffbdc62f\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") "
Oct 30 19:28:11.487641 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.487559    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\") pod \"35a6702b-966a-4813-875e-c148ffbdc62f\" (UID: \"35a6702b-966a-4813-875e-c148ffbdc62f\") "
Oct 30 19:28:11.487937 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.487917    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume" pod "35a6702b-966a-4813-875e-c148ffbdc62f" (UID: "35a6702b-966a-4813-875e-c148ffbdc62f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:11.490499 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.490464    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf" pod "35a6702b-966a-4813-875e-c148ffbdc62f" (UID: "35a6702b-966a-4813-875e-c148ffbdc62f"). InnerVolumeSpecName "kube-api-access-qmnwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:11.588824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.588787    5065 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a6702b-966a-4813-875e-c148ffbdc62f-plugins-config-volume\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.588824 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.588822    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnwf\" (UniqueName: \"kubernetes.io/projected/35a6702b-966a-4813-875e-c148ffbdc62f-kube-api-access-qmnwf\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.842495 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:11.842417    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsqg_af0fa85d-ed63-4c01-9a04-0ee25ca545a1/manager/0.log"
Oct 30 19:28:12.149169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.149071    5065 generic.go:356] "Generic (PLEG): container finished" podID="35a6702b-966a-4813-875e-c148ffbdc62f" containerID="7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef" exitCode=0
Oct 30 19:28:12.149169 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.149149    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"
Oct 30 19:28:12.149361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.149172    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" event={"ID":"35a6702b-966a-4813-875e-c148ffbdc62f","Type":"ContainerDied","Data":"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"}
Oct 30 19:28:12.149361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.149238    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw" event={"ID":"35a6702b-966a-4813-875e-c148ffbdc62f","Type":"ContainerDied","Data":"ce1f8b805414a50b62aff48a3673248fd69cc0635e94c4d392eb83eb541c5eff"}
Oct 30 19:28:12.149361 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.149271    5065 scope.go:122] "RemoveContainer" containerID="7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"
Oct 30 19:28:12.160509 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.160491    5065 scope.go:122] "RemoveContainer" containerID="7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"
Oct 30 19:28:12.160799 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:28:12.160777    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef\": container with ID starting with 7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef not found: ID does not exist" containerID="7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"
Oct 30 19:28:12.160909 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.160805    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef"} err="failed to get container status \"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef\": rpc error: code = NotFound desc = could not find container \"7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef\": container with ID starting with 7c3d247067700050fc9f49ecf724fcfc82b48384cf790037cc074ad229cd59ef not found: ID does not exist"
Oct 30 19:28:12.171726 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.171697    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 19:28:12.178026 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.178004    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8a92cc-569895d6dc-w8nhw"]
Oct 30 19:28:12.313029 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:12.313004    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f8r52_29e00d0c-30b8-45fd-9b1a-a26e6bed0594/manager/0.log"
Oct 30 19:28:13.105582 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:13.105557    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:28:13.298620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:13.298597    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:28:13.498112 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:13.498029    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:28:13.888589 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:13.888556    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" path="/var/lib/kubelet/pods/35a6702b-966a-4813-875e-c148ffbdc62f/volumes"
Oct 30 19:28:14.709448 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.709424    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 19:28:14.816196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816167    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816211    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816242    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816264    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816296    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816323    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816358 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816357    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.816779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816638    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:14.816779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816631    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:14.816779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816659    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:14.816779 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816754    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.817020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816856    5065 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\") pod \"2976d237-7f8d-48ae-9141-fbaadb689aa6\" (UID: \"2976d237-7f8d-48ae-9141-fbaadb689aa6\") "
Oct 30 19:28:14.817020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816857    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:14.817020 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.816944    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:14.817263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.817243    5065 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.817263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.817263    5065 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-credential-socket\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.817377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.817273    5065 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-data\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.817377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.817281    5065 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2976d237-7f8d-48ae-9141-fbaadb689aa6-istiod-ca-cert\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.817377 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.817291    5065 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-workload-certs\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.818620 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.818591    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:14.819010 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.818987    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:14.819216 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.819183    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:28:14.819285 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.819193    5065 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9" pod "2976d237-7f8d-48ae-9141-fbaadb689aa6" (UID: "2976d237-7f8d-48ae-9141-fbaadb689aa6"). InnerVolumeSpecName "kube-api-access-tksk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:14.892844 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.892818    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:28:14.918458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.918437    5065 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tksk9\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-kube-api-access-tksk9\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.918458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.918457    5065 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-token\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.918597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.918466    5065 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-podinfo\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:14.918597 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:14.918475    5065 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2976d237-7f8d-48ae-9141-fbaadb689aa6-istio-envoy\") on node \"ip-10-0-131-155.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:15.105019 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.104962    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:28:15.172986 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.172959    5065 generic.go:356] "Generic (PLEG): container finished" podID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerID="e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538" exitCode=0
Oct 30 19:28:15.173166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.173030    5065 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"
Oct 30 19:28:15.173166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.173029    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" event={"ID":"2976d237-7f8d-48ae-9141-fbaadb689aa6","Type":"ContainerDied","Data":"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"}
Oct 30 19:28:15.173166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.173148    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5" event={"ID":"2976d237-7f8d-48ae-9141-fbaadb689aa6","Type":"ContainerDied","Data":"da70be5678a4148f9926118ec187294797678596725a8de33785d938617b8752"}
Oct 30 19:28:15.173284 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.173178    5065 scope.go:122] "RemoveContainer" containerID="e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"
Oct 30 19:28:15.184364 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.184345    5065 scope.go:122] "RemoveContainer" containerID="e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"
Oct 30 19:28:15.184638 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:28:15.184618    5065 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538\": container with ID starting with e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538 not found: ID does not exist" containerID="e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"
Oct 30 19:28:15.184689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.184647    5065 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538"} err="failed to get container status \"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538\": rpc error: code = NotFound desc = could not find container \"e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538\": container with ID starting with e3e91202b28f4ab95d2707b724b04d5d2759fefd37a67bd64dda487c35711538 not found: ID does not exist"
Oct 30 19:28:15.198894 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.198873    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 19:28:15.203999 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.203977    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lfvl5"]
Oct 30 19:28:15.888893 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:15.888858    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" path="/var/lib/kubelet/pods/2976d237-7f8d-48ae-9141-fbaadb689aa6/volumes"
Oct 30 19:28:16.082689 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:16.082660    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf474c468-bdpsm_34502ce6-c71e-47ad-b608-7b50a8214301/authorino/0.log"
Oct 30 19:28:16.981593 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:16.981566    5065 scope.go:122] "RemoveContainer" containerID="34cf35057ab2083945953386b1ff370244af0bc8e53e01d4a6976f41b41ea7e0"
Oct 30 19:28:16.991343 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:16.991320    5065 scope.go:122] "RemoveContainer" containerID="61a3f1dcfdb5f76f5480da20beeda7b38dd367aa1bf6e0d5f22ee7958bb2757d"
Oct 30 19:28:17.000579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.000559    5065 scope.go:122] "RemoveContainer" containerID="74c0da0035984ae00995ae64d0da377ac6af9a7307edc27f29418bdc241b73cc"
Oct 30 19:28:17.009323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.009305    5065 scope.go:122] "RemoveContainer" containerID="3d5d70f687b4a4a9a5b609aba410141b8da1d2c02ed4a2affee95f3816136199"
Oct 30 19:28:17.019200 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.019179    5065 scope.go:122] "RemoveContainer" containerID="4244bbce1d870b12fc8b1fe9727dc6dd0e3ce8ba078127f4bf6b1a6dacf450c9"
Oct 30 19:28:17.027587 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.027570    5065 scope.go:122] "RemoveContainer" containerID="f427c0d6c4fcc91e0a421259e54fece6de5436e59a8d1ccc2f7e2e3b4c82393d"
Oct 30 19:28:17.035774 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.035758    5065 scope.go:122] "RemoveContainer" containerID="613adb1a695ceebbf8f2a6302e4ed5d99c7c2859ced488088a722ffd0afd3d18"
Oct 30 19:28:17.043858 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.043840    5065 scope.go:122] "RemoveContainer" containerID="937da0ba203fb3f3b1f523c6e684eb17a2256941d98b053eb385202669170503"
Oct 30 19:28:17.051860 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.051843    5065 scope.go:122] "RemoveContainer" containerID="7d65589b1318bc5ab764cfd228bb1e568135469719400833a99cfd2d2ce71962"
Oct 30 19:28:17.059733 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.059719    5065 scope.go:122] "RemoveContainer" containerID="6e0476e4e22479f8f946d6592712e799b5b1a2ed263e51dedfeedf5d4d850e49"
Oct 30 19:28:17.067821 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.067800    5065 scope.go:122] "RemoveContainer" containerID="fcc80dba6b26fc8645ff3d7c8c77e1ded201646e39746a0b7956c4041e5ab7de"
Oct 30 19:28:17.076309 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:17.076289    5065 scope.go:122] "RemoveContainer" containerID="59786920d986b9b0c70264b677b0bfe76441144990f9b106215e600591de6e7b"
Oct 30 19:28:23.056388 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:28:23.056353    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:28:27.103810 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:27.103776    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:28:27.308531 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:27.308504    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:28:27.495875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:27.495787    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:28:28.862185 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:28.862156    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:28:29.072835 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:29.072796    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:28:30.240711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:30.240674    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-174ee13d-nkxqt_af31fba6-39b1-4c2f-81d3-344cbea4ebf9/revoke-keys/0.log"
Oct 30 19:28:30.423579 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:30.423547    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-443a1993-rwj5h_d6ddf00c-585b-4815-af86-67ddccb20b85/revoke-keys/0.log"
Oct 30 19:28:30.775413 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:30.775376    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5ee29955-l4nz9_98b134a8-8eef-4dc2-a0e0-3ca5b0c9ed2e/revoke-keys/0.log"
Oct 30 19:28:30.959323 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:30.959298    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a292131d-4wzdl_371d7d60-421f-4d8e-b4ad-7415c278d390/revoke-keys/0.log"
Oct 30 19:28:31.145546 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:31.145473    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa13959e-z2nzq_beacfd19-775d-43e6-ba57-22d4e81991e2/revoke-keys/0.log"
Oct 30 19:28:31.328234 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:31.328204    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-t4gbc_46c4e3cc-8401-41d5-83f8-7ea5632ef388/postgres/0.log"
Oct 30 19:28:32.884384 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:32.884348    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf474c468-bdpsm_34502ce6-c71e-47ad-b608-7b50a8214301/authorino/0.log"
Oct 30 19:28:33.282922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:33.282894    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dpdm4_069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4/manager/0.log"
Oct 30 19:28:34.106845 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:34.106811    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsqg_af0fa85d-ed63-4c01-9a04-0ee25ca545a1/manager/0.log"
Oct 30 19:28:34.541123 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:34.541071    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f8r52_29e00d0c-30b8-45fd-9b1a-a26e6bed0594/manager/0.log"
Oct 30 19:28:35.702907 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:35.702876    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q46vl_1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3/discovery/0.log"
Oct 30 19:28:35.884485 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:35.884453    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-7k7ts_9a3778ff-4333-47d7-a887-a8a669a192d8/kube-auth-proxy/0.log"
Oct 30 19:28:36.619460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:36.619424    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-p8smm_ea311f74-4221-4d7a-a3da-bc4a30848044/istio-proxy/0.log"
Oct 30 19:28:37.171515 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:37.171481    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-v8w6k_ca1f838b-93cf-425f-b9ac-232488fca33e/istio-proxy/0.log"
Oct 30 19:28:37.695014 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:37.694984    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 19:28:37.905420 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:37.905389    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f55855fc-ctkk9_3bc815a3-5bd4-4ec0-b417-29f36198d1be/router/0.log"
Oct 30 19:28:38.870016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:38.869983    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tss5n_b0bcbabc-ea43-41cc-b40d-f0cfefae6f0a/main/0.log"
Oct 30 19:28:39.049496 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:39.049456    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2cjgq_9ba78a05-9a5a-466a-bc96-0628a262466b/main/0.log"
Oct 30 19:28:39.419071 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:39.419046    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jppv6_dfeaf827-b7cd-4368-ae2c-b553d38301d0/main/0.log"
Oct 30 19:28:39.959782 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:39.959756    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5tvt7_fa456d21-114b-4b81-93b9-64f94cd6e963/main/0.log"
Oct 30 19:28:48.034875 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:48.034843    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"]
Oct 30 19:28:48.038576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:48.038546    5065 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"]
Oct 30 19:28:48.041141 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:48.041098    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a292131d-4wzdl"]
Oct 30 19:28:48.043301 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:48.043278    5065 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa13959e-z2nzq"]
Oct 30 19:28:49.901961 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:49.901916    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371d7d60-421f-4d8e-b4ad-7415c278d390" path="/var/lib/kubelet/pods/371d7d60-421f-4d8e-b4ad-7415c278d390/volumes"
Oct 30 19:28:49.902447 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:49.902390    5065 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beacfd19-775d-43e6-ba57-22d4e81991e2" path="/var/lib/kubelet/pods/beacfd19-775d-43e6-ba57-22d4e81991e2/volumes"
Oct 30 19:28:53.059676 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:28:53.059632    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:28:55.011708 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.011674    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lg2zm_afe4c4f3-230a-4545-844b-fd72139f8e6d/global-pull-secret-syncer/0.log"
Oct 30 19:28:55.208272 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.208245    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hghfk_200b118b-598b-4885-a4d6-f570d78aa4a7/connection-checker/0.log"
Oct 30 19:28:55.240686 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.240655    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vczpg_a927c7bb-ddf0-4f1d-aafd-34b7b3881fc4/connection-checker/0.log"
Oct 30 19:28:55.276497 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.276446    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-whf9r_3f4b50cf-c2e9-43b7-b6e5-9fc9f3e16562/connection-checker/0.log"
Oct 30 19:28:55.462016 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.461993    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nhsc4_3fb2a473-98fc-4918-a07e-fb7d4edfddb8/konnectivity-agent/0.log"
Oct 30 19:28:55.494273 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:28:55.494254    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-155.us-east-2.compute.internal_dabecda5632cef959a155a21c4c95263/haproxy/0.log"
Oct 30 19:29:03.098459 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:03.098427    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf474c468-bdpsm_34502ce6-c71e-47ad-b608-7b50a8214301/authorino/0.log"
Oct 30 19:29:03.166503 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:03.166466    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dpdm4_069f3fdc-c1ae-47d3-ad80-9b1b169bc3f4/manager/0.log"
Oct 30 19:29:03.617196 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:03.617167    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-dwsqg_af0fa85d-ed63-4c01-9a04-0ee25ca545a1/manager/0.log"
Oct 30 19:29:03.780213 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:03.780183    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f8r52_29e00d0c-30b8-45fd-9b1a-a26e6bed0594/manager/0.log"
Oct 30 19:29:05.755552 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:05.755524    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-lf2ct_263e2d58-6476-493e-9de0-319a8ed0c5f7/cluster-monitoring-operator/0.log"
Oct 30 19:29:05.973324 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:05.973303    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9g4zt_c334df43-4158-434b-bd57-ff8119a576a8/node-exporter/0.log"
Oct 30 19:29:06.008525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:06.008497    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9g4zt_c334df43-4158-434b-bd57-ff8119a576a8/kube-rbac-proxy/0.log"
Oct 30 19:29:06.040195 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:06.040168    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9g4zt_c334df43-4158-434b-bd57-ff8119a576a8/init-textfile/0.log"
Oct 30 19:29:07.073696 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.073664    5065 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"]
Oct 30 19:29:07.074329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074302    5065 cpu_manager.go:414] "containerMap: removing container" podUID="d6ddf00c-585b-4815-af86-67ddccb20b85" containerName="revoke-keys"
Oct 30 19:29:07.074329 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074330    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ddf00c-585b-4815-af86-67ddccb20b85" containerName="revoke-keys"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074350    5065 cpu_manager.go:414] "containerMap: removing container" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074358    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074377    5065 cpu_manager.go:414] "containerMap: removing container" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074385    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074411    5065 cpu_manager.go:414] "containerMap: removing container" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy"
Oct 30 19:29:07.074498 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074418    5065 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy"
Oct 30 19:29:07.074822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074523    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ddf00c-585b-4815-af86-67ddccb20b85" containerName="revoke-keys"
Oct 30 19:29:07.074822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074537    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c74c55-4e95-4d65-bd8d-b36572307f6c" containerName="main"
Oct 30 19:29:07.074822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074549    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a6702b-966a-4813-875e-c148ffbdc62f" containerName="payload-pre-processing"
Oct 30 19:29:07.074822 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.074561    5065 memory_manager.go:365] "RemoveStaleState removing state" podUID="2976d237-7f8d-48ae-9141-fbaadb689aa6" containerName="istio-proxy"
Oct 30 19:29:07.077867 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.077847    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.084797 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.084770    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"openshift-service-ca.crt\""
Oct 30 19:29:07.085576 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.085270    5065 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"kube-root-ca.crt\""
Oct 30 19:29:07.087956 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.087940    5065 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nwzbq\"/\"default-dockercfg-jgvfw\""
Oct 30 19:29:07.088281 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.088258    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"]
Oct 30 19:29:07.199511 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.199486    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-proc\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.199636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.199528    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzrg\" (UniqueName: \"kubernetes.io/projected/100752fc-2965-4361-a389-a8d11eb1abf4-kube-api-access-kkzrg\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.199636 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.199555    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-podres\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.199730 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.199688    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-sys\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.199783 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.199725    5065 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-lib-modules\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300657    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-proc\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300710    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzrg\" (UniqueName: \"kubernetes.io/projected/100752fc-2965-4361-a389-a8d11eb1abf4-kube-api-access-kkzrg\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300739    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-podres\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300770    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-sys\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300786    5065 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-lib-modules\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.300823 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300791    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-proc\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.301131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300867    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-podres\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.301131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300882    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-lib-modules\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.301131 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.300888    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/100752fc-2965-4361-a389-a8d11eb1abf4-sys\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.314796 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.314767    5065 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzrg\" (UniqueName: \"kubernetes.io/projected/100752fc-2965-4361-a389-a8d11eb1abf4-kube-api-access-kkzrg\") pod \"perf-node-gather-daemonset-bwzp8\" (UID: \"100752fc-2965-4361-a389-a8d11eb1abf4\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.392335 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.392280    5065 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:07.691353 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:07.691322    5065 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"]
Oct 30 19:29:07.693117 ip-10-0-131-155 kubenswrapper[5065]: W1030 19:29:07.693062    5065 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod100752fc_2965_4361_a389_a8d11eb1abf4.slice/crio-8b454862de8b71c6f240120339d4b89591e49cc76c747e5d3c557101689604ac WatchSource:0}: Error finding container 8b454862de8b71c6f240120339d4b89591e49cc76c747e5d3c557101689604ac: Status 404 returned error can't find the container with id 8b454862de8b71c6f240120339d4b89591e49cc76c747e5d3c557101689604ac
Oct 30 19:29:08.512653 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:08.512616    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8" event={"ID":"100752fc-2965-4361-a389-a8d11eb1abf4","Type":"ContainerStarted","Data":"5f6eee6d94be6166849d934b5c54a3a4be3e2a7714ee0e7c14fba6a2d6470147"}
Oct 30 19:29:08.513154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:08.512676    5065 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8" event={"ID":"100752fc-2965-4361-a389-a8d11eb1abf4","Type":"ContainerStarted","Data":"8b454862de8b71c6f240120339d4b89591e49cc76c747e5d3c557101689604ac"}
Oct 30 19:29:08.513154 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:08.512749    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:08.533376 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:08.533316    5065 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8" podStartSLOduration=1.5333047770000001 podStartE2EDuration="1.533304777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:08.532449711 +0000 UTC m=+3368.795914659" watchObservedRunningTime="2026-10-30 19:29:08.533304777 +0000 UTC m=+3368.796769725"
Oct 30 19:29:08.752201 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:08.752166    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-gjzjs_b6d938f8-d5bf-42a9-b4c5-169b0f0d6777/networking-console-plugin/0.log"
Oct 30 19:29:09.645274 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:09.645235    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/1.log"
Oct 30 19:29:09.651528 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:09.651498    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rsttd_0fab5545-eaa2-487e-890b-f21a689ee08c/console-operator/2.log"
Oct 30 19:29:11.300521 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:11.300424    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:11.314427 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:11.314396    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:11.367458 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:11.367437    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ntj76_9ce97a9d-4f14-469c-9258-12f174cf1ac4/volume-data-source-validator/0.log"
Oct 30 19:29:11.372229 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:11.372199    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:12.244172 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:12.244130    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:12.260818 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:12.260790    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:12.316701 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:12.316677    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:13.136600 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:13.136568    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8bh7_8391c80c-4407-4fc4-9053-b336be68476c/dns/0.log"
Oct 30 19:29:13.170825 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:13.170798    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8bh7_8391c80c-4407-4fc4-9053-b336be68476c/kube-rbac-proxy/0.log"
Oct 30 19:29:13.372764 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:13.372733    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-prl4g_0dc98e36-faf2-4b8a-b2b5-b1d295c01fc2/dns-node-resolver/0.log"
Oct 30 19:29:13.792837 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:13.792805    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:13.809769 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:13.809742    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:14.322516 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:14.322487    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fqpv4_2fa09131-1c47-4d63-82d2-c5174c60fd49/node-ca/0.log"
Oct 30 19:29:14.531455 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:14.531427    5065 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-bwzp8"
Oct 30 19:29:14.776711 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:14.776681    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:14.795922 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:14.795900    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:15.260675 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:15.260647    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:15.272263 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:15.272240    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:15.328300 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:15.328275    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:16.411659 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:16.411632    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:16.427635 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:16.427608    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:16.502163 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:16.502130    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:16.952357 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:16.952326    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q46vl_1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3/discovery/0.log"
Oct 30 19:29:17.194946 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.194912    5065 scope.go:122] "RemoveContainer" containerID="e8bfb8b17b3716f2a3e098c15c01ed1ca40a2d0c5c06340b0f08b1e678d3793d"
Oct 30 19:29:17.207728 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.207677    5065 scope.go:122] "RemoveContainer" containerID="adf8b2fa20d81d409fbbc35fd36426bef8cb7900129af3e644c5eb50ec10f346"
Oct 30 19:29:17.217655 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.217633    5065 scope.go:122] "RemoveContainer" containerID="6023e333939437ec74f67b6fc831c2525ab44e12de524c6f76c4f5ae8630a051"
Oct 30 19:29:17.226792 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.226774    5065 scope.go:122] "RemoveContainer" containerID="d82aa4d525a626d7a8865503c81c2b94382b0801418a6ee10abaeda8306d2808"
Oct 30 19:29:17.737657 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.737626    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-7k7ts_9a3778ff-4333-47d7-a887-a8a669a192d8/kube-auth-proxy/0.log"
Oct 30 19:29:17.841460 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.841429    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:17.893276 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.893254    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:17.986348 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:17.986326    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-p8smm_ea311f74-4221-4d7a-a3da-bc4a30848044/istio-proxy/0.log"
Oct 30 19:29:18.084671 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:18.084650    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-v8w6k_ca1f838b-93cf-425f-b9ac-232488fca33e/istio-proxy/0.log"
Oct 30 19:29:18.198667 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:18.198643    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 19:29:18.251525 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:18.251501    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f55855fc-ctkk9_3bc815a3-5bd4-4ec0-b417-29f36198d1be/router/0.log"
Oct 30 19:29:18.972603 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:18.972570    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:19.012601 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:19.012578    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:19.370326 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:19.370298    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:19.383715 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:19.383691    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:19.443009 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:19.442981    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:20.550788 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:20.550762    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f569dfc6b-gn7sj_81f025d9-65d5-4552-bcec-5766fe5f6182/manager/0.log"
Oct 30 19:29:20.578038 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:20.578006    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6g8zb_7408db0f-b1e1-4ef0-ab31-43955955d798/manager/0.log"
Oct 30 19:29:20.672079 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:20.672044    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vnk4q_33068276-152a-4e10-bce0-8b177f00d071/manager/0.log"
Oct 30 19:29:21.293917 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.293887    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrlwc_b264fe31-7f47-4667-9563-3ec98cf45e9b/serve-healthcheck-canary/0.log"
Oct 30 19:29:21.640850 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.640775    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q46vl_1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3/discovery/0.log"
Oct 30 19:29:21.684238 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.684208    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-7k7ts_9a3778ff-4333-47d7-a887-a8a669a192d8/kube-auth-proxy/0.log"
Oct 30 19:29:21.833106 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.833052    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-p8smm_ea311f74-4221-4d7a-a3da-bc4a30848044/istio-proxy/0.log"
Oct 30 19:29:21.912383 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.912330    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-v8w6k_ca1f838b-93cf-425f-b9ac-232488fca33e/istio-proxy/0.log"
Oct 30 19:29:21.912877 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.912855    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-ghhlw_3b50a449-623e-44d0-bdb9-541e633dab54/insights-operator/0.log"
Oct 30 19:29:21.914536 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.914514    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-ghhlw_3b50a449-623e-44d0-bdb9-541e633dab54/insights-operator/1.log"
Oct 30 19:29:21.980350 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:21.980318    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:22.040446 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.040426    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:22.049135 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.049118    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 19:29:22.089683 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.089664    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f55855fc-ctkk9_3bc815a3-5bd4-4ec0-b417-29f36198d1be/router/0.log"
Oct 30 19:29:22.120959 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.120941    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvfzd_f03be630-d162-45d2-8d86-6dceae239940/kube-rbac-proxy/0.log"
Oct 30 19:29:22.153432 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.153413    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvfzd_f03be630-d162-45d2-8d86-6dceae239940/exporter/0.log"
Oct 30 19:29:22.185638 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:22.185581    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvfzd_f03be630-d162-45d2-8d86-6dceae239940/extractor/0.log"
Oct 30 19:29:23.067908 ip-10-0-131-155 kubenswrapper[5065]: E1030 19:29:23.067864    5065 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hvfzd" containerName="extractor"
Oct 30 19:29:23.129166 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:23.129134    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2bnm_6a46c63a-9f3b-4668-af47-a9cc278533e9/manager/0.log"
Oct 30 19:29:23.213061 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:23.213041    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-7877dff649-lqdc8_b5963b47-ba49-4aa8-a578-f4abe50945d2/manager/0.log"
Oct 30 19:29:24.734053 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:24.734027    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q46vl_1a09bc3c-6f71-47ed-a0f2-7c2b0aaafca3/discovery/0.log"
Oct 30 19:29:24.751445 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:24.751421    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-7k7ts_9a3778ff-4333-47d7-a887-a8a669a192d8/kube-auth-proxy/0.log"
Oct 30 19:29:24.859065 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:24.859041    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-p8smm_ea311f74-4221-4d7a-a3da-bc4a30848044/istio-proxy/0.log"
Oct 30 19:29:24.916212 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:24.916189    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-v8w6k_ca1f838b-93cf-425f-b9ac-232488fca33e/istio-proxy/0.log"
Oct 30 19:29:24.988880 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:24.988818    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wzss6_015a612c-9ab3-4629-adec-475bca8e56e4/payload-processing/0.log"
Oct 30 19:29:25.017991 ip-10-0-131-155 kubenswrapper[5065]: I1030 19:29:25.017968    5065 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f55855fc-ctkk9_3bc815a3-5bd4-4ec0-b417-29f36198d1be/router/0.log"